2024-03-29T10:35:27.912 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:35:27.912 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656 --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656/orig.config.yaml 2024-03-29T10:35:27.947 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:35:28.026 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622656' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzB2nmx/jDNQL6ad35f2kCeylwaq4F/KlMlDoy0UNN5CCNeOuTbKQ3wP8tds7fvlFZfqRPeNnBzFN4JMvVnTcM= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcegcc0yDoT9QN1F1Kn+v508+jLs+9MnP5xfCQMWLaSXylIex7wd+H/dWc3zhPQGh+3o98lkHn9JWeyby1HAZo= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:35:28.026 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:35:28.027 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:35:28.027 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:35:28.028 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:35:28.053 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:35:28.053 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:35:28.053 INFO:teuthology.packaging:ref: None 2024-03-29T10:35:28.053 INFO:teuthology.packaging:tag: None 2024-03-29T10:35:28.053 INFO:teuthology.packaging:branch: main 2024-03-29T10:35:28.054 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:35:28.054 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:35:28.340 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:35:28.341 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:35:28.348 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:35:28.348 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:35:28.372 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:35:28.388 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:35:28.407 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:35:28.429 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:30:37.880496', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzB2nmx/jDNQL6ad35f2kCeylwaq4F/KlMlDoy0UNN5CCNeOuTbKQ3wP8tds7fvlFZfqRPeNnBzFN4JMvVnTcM='} 2024-03-29T10:35:28.446 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:30:37.879140', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcegcc0yDoT9QN1F1Kn+v508+jLs+9MnP5xfCQMWLaSXylIex7wd+H/dWc3zhPQGh+3o98lkHn9JWeyby1HAZo='} 2024-03-29T10:35:28.447 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:35:28.455 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:35:28.455 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:35:28.455 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:35:28.537 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe2726370d0>, signals=[15]) 2024-03-29T10:35:28.538 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:35:28.546 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:35:28.547 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T10:35:28.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:35:28.655 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T10:35:28.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:35:28.744 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:35:28.763 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-03-29T10:35:28.781 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-03-29T10:35:28.781 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-03-29T10:35:28.838 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Stream" 2024-03-29T10:35:28.838 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="9" 2024-03-29T10:35:28.838 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2024-03-29T10:35:28.838 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:35:28.838 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="9" 2024-03-29T10:35:28.838 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:35:28.838 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:35:28.838 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2024-03-29T10:35:28.838 INFO:teuthology.orchestra.run.smithi005.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:35:28.839 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:35:28.839 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:35:28.839 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:35:28.839 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:35:28.839 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:35:28.839 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-03-29T10:35:28.861 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-03-29T10:35:28.880 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-03-29T10:35:28.880 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-03-29T10:35:28.939 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-03-29T10:35:28.939 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-03-29T10:35:28.939 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-03-29T10:35:28.939 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:35:28.939 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-03-29T10:35:28.939 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:35:28.940 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:35:28.940 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-03-29T10:35:28.940 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:35:28.940 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:35:28.940 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:35:28.940 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:35:28.940 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:35:28.940 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:35:28.940 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-03-29T10:35:28.961 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:35:28.979 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:35:29.060 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:35:29.060 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:35:29.063 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:35:29.080 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:35:29.095 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:35:29.095 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:35:29.122 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:35:29.148 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:35:29.213 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:35:29.232 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:35:29.233 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:35:29.233 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:35:29.233 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:35:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:35:29.251 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-03-29T10:35:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:35:29.252 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-03-29T10:35:29.310 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:35:29.310 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-361.el9.x86_64 2024-03-29T10:35:29.310 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-03-29T10:35:29.311 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:35:29.312 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.14.0-361.el9.x86_64 2024-03-29T10:35:29.313 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum install -y kernel 2024-03-29T10:35:31.733 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 8.4 MB/s | 6.7 MB 00:00 2024-03-29T10:35:32.315 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - AppStream 4.7 MB/s | 6.7 MB 00:01 2024-03-29T10:35:33.166 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-29T10:35:33.790 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-29T10:35:34.206 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T10:35:35.846 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-29T10:35:36.024 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-29T10:35:38.158 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-29T10:35:39.950 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T10:35:41.751 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-29T10:35:43.198 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-03-29T10:35:44.477 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 21 MB 00:02 2024-03-29T10:35:44.665 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-29T10:35:47.867 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 21 MB 00:03 2024-03-29T10:35:49.460 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-03-29T10:35:51.691 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:35:51.692 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:35:51.781 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:35:51.793 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:35:51.793 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:35:51.793 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:35:51.793 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:35:51.793 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:35:51.794 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-29T10:35:51.794 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:35:51.794 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:35:51.794 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:35:51.794 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:35:51.794 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:35:51.794 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:35:51.794 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-03-29T10:35:51.794 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:35:51.795 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 76 M 2024-03-29T10:35:51.795 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 123 M 2024-03-29T10:35:51.795 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:35:51.953 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T10:35:52.786 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 15 MB/s | 15 MB 00:00 2024-03-29T10:35:53.067 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-03-29T10:35:53.311 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-03-29T10:35:53.778 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T10:35:53.781 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:53.781 INFO:teuthology.orchestra.run.smithi177.stdout:Total 38 MB/s | 76 MB 00:01 2024-03-29T10:35:53.782 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:35:54.129 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:35:54.129 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:35:54.979 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:35:54.979 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:35:55.465 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:35:55.465 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:35:55.557 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:35:55.565 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout:Install 4 Packages 2024-03-29T10:35:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:35:55.567 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 76 M 2024-03-29T10:35:55.567 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 123 M 2024-03-29T10:35:55.567 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:35:55.901 INFO:teuthology.orchestra.run.smithi005.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 574 kB/s | 192 kB 00:00 2024-03-29T10:35:57.109 INFO:teuthology.orchestra.run.smithi005.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 9.8 MB/s | 15 MB 00:01 2024-03-29T10:35:57.260 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:35:57.718 INFO:teuthology.orchestra.run.smithi005.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T10:35:57.866 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:35:57.946 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:35:58.043 INFO:teuthology.orchestra.run.smithi005.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 14 MB/s | 34 MB 00:02 2024-03-29T10:35:58.047 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:58.047 INFO:teuthology.orchestra.run.smithi005.stdout:Total 31 MB/s | 76 MB 00:02 2024-03-29T10:35:58.047 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:35:58.469 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:35:58.469 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:35:59.369 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:35:59.369 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:36:00.672 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:00.823 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:01.831 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:36:02.533 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:36:02.618 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:05.531 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:05.539 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:05.564 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:10.461 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:10.735 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:10.778 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:15.883 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:39.740 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:39.740 INFO:teuthology.orchestra.run.smithi177.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:36:39.740 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:36:39.784 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:40.182 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:40.183 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:36:40.183 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:40.183 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:40.591 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:40.592 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:36:40.592 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:36:40.592 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:36:40.592 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:36:40.592 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:36:40.592 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:36:40.592 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:36:40.592 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:36:40.919 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:36:41.027 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:36:41.027 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:36:41.027 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:36:41.028 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:36:41.028 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:36:41.028 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:36:41.028 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:36:41.028 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:36:41.108 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:36:41.108 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:36:41.108 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-03-29T10:36:41.151 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-03-29T10:36:41.151 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-03-29T10:36:41.151 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2024-03-29T10:36:41.151 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 68.el9 2024-03-29T10:36:41.151 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-03-29T10:36:41.151 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:36:41.151 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2024-03-29T10:36:41.151 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8264254 2024-03-29T10:36:41.152 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-03-29T10:36:41.152 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:36:41.152 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:36:41.152 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:36:41.152 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:36:41.152 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2024-03-29T10:36:41.152 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-03-29T10:36:41.152 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:36:41.152 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-03-29T10:36:41.152 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-03-29T10:36:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:36:41.153 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:36:41.153 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:36:41.153 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:36:41.153 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-03-29T10:36:41.153 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:36:41.153 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:36:41.154 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:36:41.154 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:36:41.154 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:36:41.808 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-03-29T10:36:43.162 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:36:43.189 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-03-29T10:36:43.191 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-03-29T10:36:43.206 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.LtnddhulOY 2024-03-29T10:36:43.206 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LtnddhulOY 2024-03-29T10:36:43.275 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.LtnddhulOY 2024-03-29T10:36:43.416 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.LtnddhulOY is 6KB 2024-03-29T10:36:43.465 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.LtnddhulOY 2024-03-29T10:36:43.479 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:36:43.553 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:36:43.553 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:36:43.553 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:36:43.553 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:36:43.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:36:43.715 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-03-29T10:36:46.863 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:46.863 INFO:teuthology.orchestra.run.smithi005.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:36:46.863 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:36:46.908 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:47.273 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:47.274 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:36:47.274 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:47.274 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:47.691 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:36:47.691 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:36:47.691 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:36:47.691 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:36:47.691 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:36:47.691 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:36:47.691 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:36:47.691 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:36:47.691 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:36:48.052 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:36:48.206 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:36:48.206 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:36:48.206 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:36:48.206 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:36:48.207 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:36:48.207 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:36:48.207 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:36:48.207 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:36:48.290 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:36:48.291 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:36:48.291 DEBUG:teuthology.orchestra.run.smithi005:> sudo rpm -qi grub2-tools 2024-03-29T10:36:48.332 INFO:teuthology.orchestra.run.smithi005.stdout:Name : grub2-tools 2024-03-29T10:36:48.332 INFO:teuthology.orchestra.run.smithi005.stdout:Epoch : 1 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Version : 2.06 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Release : 68.el9 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: x86_64 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Group : Unspecified 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Size : 8264254 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:License : GPLv3+ 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Packager : builder@centos.org 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:Vendor : CentOS 2024-03-29T10:36:48.333 INFO:teuthology.orchestra.run.smithi005.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:36:48.334 INFO:teuthology.orchestra.run.smithi005.stdout:Summary : Support tools for GRUB. 2024-03-29T10:36:48.334 INFO:teuthology.orchestra.run.smithi005.stdout:Description : 2024-03-29T10:36:48.334 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:36:48.334 INFO:teuthology.orchestra.run.smithi005.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:36:48.334 INFO:teuthology.orchestra.run.smithi005.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:36:48.334 INFO:teuthology.orchestra.run.smithi005.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:36:48.334 INFO:teuthology.orchestra.run.smithi005.stdout:hardware devices. 2024-03-29T10:36:48.334 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:36:48.334 INFO:teuthology.orchestra.run.smithi005.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:36:48.335 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:36:48.335 INFO:teuthology.task.kernel:Updating grub on smithi005 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:36:48.335 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:36:48.944 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-03-29T10:36:50.295 INFO:teuthology.orchestra.run.smithi005.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:36:50.336 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-03-29T10:36:50.338 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-03-29T10:36:50.353 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.tpKnEeTUR9 2024-03-29T10:36:50.354 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tpKnEeTUR9 2024-03-29T10:36:50.426 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.tpKnEeTUR9 2024-03-29T10:36:50.573 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.tpKnEeTUR9 is 6KB 2024-03-29T10:36:50.622 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.tpKnEeTUR9 2024-03-29T10:36:50.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:36:50.706 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:36:50.706 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:36:50.706 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:36:50.706 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:36:50.707 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:36:50.860 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-03-29T10:37:13.738 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:37:13.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-29T10:37:13.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:37:20.870 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:37:20.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-29T10:37:20.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:37:48.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-29T10:37:57.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-29T10:37:57.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:38:15.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-29T10:38:20.874 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:38:27.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-29T10:38:27.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:38:27.925 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-29T10:38:28.313 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-29T10:38:28.313 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:38:28.314 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-03-29T10:38:28.328 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:38:28.328 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:38:28.329 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:38:28.329 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:38:29.329 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:38:29.329 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:38:29.390 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-03-29T10:38:29.425 DEBUG:teuthology.parallel:result is None 2024-03-29T10:38:29.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-29T10:38:29.876 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:38:30.322 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-03-29T10:38:30.710 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-29T10:38:30.711 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:38:30.711 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-03-29T10:38:30.724 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:38:30.725 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:38:30.725 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:38:30.725 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:38:31.726 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:38:31.726 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:38:31.800 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-03-29T10:38:31.839 DEBUG:teuthology.parallel:result is None 2024-03-29T10:38:31.840 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:38:31.850 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:38:31.850 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:38:31.853 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:38:31.867 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:38:31.898 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:38:31.927 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:38:31.928 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:38:31.930 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:38:31.961 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:38:31.968 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:38:31.969 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:38:31.981 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:38:32.010 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:38:32.023 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:38:32.026 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:38:32.039 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:38:32.041 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:38:32.048 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:38:32.048 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:38:32.064 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:38:32.107 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:38:32.115 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:38:32.115 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:38:32.127 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:38:32.159 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:38:32.216 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:38:32.261 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:38:32.341 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:38:32.380 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:38:32.380 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:38:32.444 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:38:32.484 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:38:32.540 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:38:32.620 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:38:32.660 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:38:32.660 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:38:32.724 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-03-29T10:38:32.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-03-29T10:38:32.774 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:38:32.815 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:38:33.305 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:38:33.313 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:38:33.313 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:38:33.339 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:38:33.372 DEBUG:teuthology.orchestra.run.smithi005:> sudo service auditd rotate 2024-03-29T10:38:33.437 INFO:teuthology.orchestra.run.smithi005.stdout:Rotating logs: 2024-03-29T10:38:33.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-03-29T10:38:33.496 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-03-29T10:38:33.498 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:38:33.498 DEBUG:teuthology.orchestra.run.smithi005:> /usr/sbin/getenforce 2024-03-29T10:38:33.541 INFO:teuthology.orchestra.run.smithi005.stdout:Permissive 2024-03-29T10:38:33.542 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-03-29T10:38:33.566 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-03-29T10:38:33.566 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi005.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:38:33.567 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:38:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:38:33.594 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:38:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:38:33.630 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:38:33.631 DEBUG:teuthology.orchestra.run.smithi005:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:38:33.710 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:38:33.743 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:38:33.753 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T10:38:33.758 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T10:38:33.883 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:38:33.901 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:38:33.903 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:42:49.104 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-03-29T10:42:49.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-29T10:42:49.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:49.201 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-03-29T10:42:49.277 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-29T10:42:49.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-29T10:42:49.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:49.365 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-29T10:42:49.444 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-29T10:42:49.445 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:42:49.456 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:42:49.457 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:42:49.457 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:42:49.460 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:42:49.460 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:42:49.490 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:42:49.507 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:42:49.515 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:42:49.531 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:42:49.537 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: ntpd: command not found 2024-03-29T10:42:49.551 INFO:teuthology.orchestra.run.smithi005.stdout:506 Cannot talk to daemon 2024-03-29T10:42:49.560 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-03-29T10:42:49.568 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:42:49.573 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-03-29T10:42:49.584 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:42:49.590 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:42:49.606 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:42:49.652 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:42:49.656 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:42:49.656 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-03-29T10:42:49.656 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:42:49.656 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:42:49.656 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:42:49.656 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:42:49.660 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:42:49.663 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:42:49.664 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-29T10:42:49.664 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:42:49.664 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:42:49.664 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:42:49.664 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:42:49.665 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:42:49.673 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:42:49.673 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:42:49.673 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:42:49.673 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:42:49.679 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:42:49.679 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:42:49.680 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:42:49.680 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:42:49.682 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:42:49.683 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:42:49.902 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:42:49.902 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:42:49.910 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:42:49.910 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:42:50.013 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:42:50.013 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:42:50.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:42:50.023 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:42:50.023 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:42:50.023 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:42:50.042 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:42:50.043 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:42:50.052 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:42:50.052 DEBUG:teuthology.orchestra.run.smithi005:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:42:50.110 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:42:50.121 DEBUG:teuthology.orchestra.run.smithi005:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:42:50.190 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:42:50.201 DEBUG:teuthology.orchestra.run.smithi005:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:42:50.257 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-03-29T10:42:50.259 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-03-29T10:42:50.267 INFO:teuthology.orchestra.run.smithi005.stdout:check_obsoletes = 1 2024-03-29T10:42:50.269 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum clean all 2024-03-29T10:42:50.498 INFO:teuthology.orchestra.run.smithi177.stdout:74 files removed 2024-03-29T10:42:50.515 INFO:teuthology.orchestra.run.smithi005.stdout:74 files removed 2024-03-29T10:42:50.525 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw 2024-03-29T10:42:50.544 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-radosgw 2024-03-29T10:42:51.228 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 204 kB/s | 82 kB 00:00 2024-03-29T10:42:51.229 INFO:teuthology.orchestra.run.smithi005.stdout:ceph packages for x86_64 216 kB/s | 82 kB 00:00 2024-03-29T10:42:51.587 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-29T10:42:51.604 INFO:teuthology.orchestra.run.smithi005.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-29T10:42:51.963 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-03-29T10:42:52.013 INFO:teuthology.orchestra.run.smithi005.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-29T10:42:52.348 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-03-29T10:42:52.435 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T10:42:53.799 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-29T10:42:54.002 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-03-29T10:42:54.900 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T10:42:59.324 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-03-29T10:43:03.333 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - BaseOS 915 kB/s | 8.0 MB 00:08 2024-03-29T10:43:03.475 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.1 MB 00:00 2024-03-29T10:43:05.117 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-29T10:43:05.952 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-03-29T10:43:10.918 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-29T10:43:13.553 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options 2024-03-29T10:43:13.563 INFO:teuthology.orchestra.run.smithi177.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:43:13.564 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:43:13.565 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-03-29T10:43:13.566 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:43:13.566 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:13.566 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:43:13.566 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:43:13.566 INFO:teuthology.orchestra.run.smithi177.stdout:Install 30 Packages 2024-03-29T10:43:13.566 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 2 Packages 2024-03-29T10:43:13.566 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:13.567 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 58 M 2024-03-29T10:43:13.567 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:43:15.169 INFO:teuthology.orchestra.run.smithi177.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 7.8 MB/s | 9.7 MB 00:01 2024-03-29T10:43:15.203 INFO:teuthology.orchestra.run.smithi177.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 763 kB/s | 25 kB 00:00 2024-03-29T10:43:15.345 INFO:teuthology.orchestra.run.smithi177.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.1 MB/s | 742 kB 00:00 2024-03-29T10:43:15.553 INFO:teuthology.orchestra.run.smithi177.stdout:(4/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 12 MB/s | 20 MB 00:01 2024-03-29T10:43:15.587 INFO:teuthology.orchestra.run.smithi177.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.0 MB/s | 498 kB 00:00 2024-03-29T10:43:15.654 INFO:teuthology.orchestra.run.smithi177.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 678 kB/s | 45 kB 00:00 2024-03-29T10:43:15.773 INFO:teuthology.orchestra.run.smithi177.stdout:(7/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.9 MB/s | 5.3 MB 00:01 2024-03-29T10:43:15.798 INFO:teuthology.orchestra.run.smithi177.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 911 kB/s | 131 kB 00:00 2024-03-29T10:43:15.873 INFO:teuthology.orchestra.run.smithi177.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 16 MB/s | 5.1 MB 00:00 2024-03-29T10:43:15.899 INFO:teuthology.orchestra.run.smithi177.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 1.3 MB/s | 162 kB 00:00 2024-03-29T10:43:15.924 INFO:teuthology.orchestra.run.smithi177.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.5 MB/s | 320 kB 00:00 2024-03-29T10:43:15.949 INFO:teuthology.orchestra.run.smithi177.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-29T10:43:15.974 INFO:teuthology.orchestra.run.smithi177.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 99 kB 00:00 2024-03-29T10:43:16.116 INFO:teuthology.orchestra.run.smithi177.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 551 kB/s | 106 kB 00:00 2024-03-29T10:43:16.141 INFO:teuthology.orchestra.run.smithi177.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 829 kB/s | 159 kB 00:00 2024-03-29T10:43:16.158 INFO:teuthology.orchestra.run.smithi177.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 335 kB/s | 44 kB 00:00 2024-03-29T10:43:16.259 INFO:teuthology.orchestra.run.smithi177.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 661 kB 00:00 2024-03-29T10:43:16.275 INFO:teuthology.orchestra.run.smithi177.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 353 kB/s | 41 kB 00:00 2024-03-29T10:43:16.301 INFO:teuthology.orchestra.run.smithi177.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 282 kB 00:00 2024-03-29T10:43:16.326 INFO:teuthology.orchestra.run.smithi177.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T10:43:16.343 INFO:teuthology.orchestra.run.smithi177.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 524 kB/s | 35 kB 00:00 2024-03-29T10:43:16.361 INFO:teuthology.orchestra.run.smithi177.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 792 kB/s | 46 kB 00:00 2024-03-29T10:43:16.635 INFO:teuthology.orchestra.run.smithi177.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 94 kB/s | 26 kB 00:00 2024-03-29T10:43:16.835 INFO:teuthology.orchestra.run.smithi177.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 243 kB/s | 49 kB 00:00 2024-03-29T10:43:16.919 INFO:teuthology.orchestra.run.smithi177.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 537 kB/s | 309 kB 00:00 2024-03-29T10:43:17.044 INFO:teuthology.orchestra.run.smithi177.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 323 kB/s | 67 kB 00:00 2024-03-29T10:43:17.186 INFO:teuthology.orchestra.run.smithi177.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-03-29T10:43:17.245 INFO:teuthology.orchestra.run.smithi177.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 957 kB/s | 192 kB 00:00 2024-03-29T10:43:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:00 2024-03-29T10:43:17.403 INFO:teuthology.orchestra.run.smithi177.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-03-29T10:43:17.620 INFO:teuthology.orchestra.run.smithi177.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-29T10:43:17.854 INFO:teuthology.orchestra.run.smithi177.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 5.5 MB/s | 3.4 MB 00:00 2024-03-29T10:43:17.858 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:17.859 INFO:teuthology.orchestra.run.smithi177.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-29T10:43:17.859 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:43:17.897 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:43:17.898 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:43:18.396 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:43:18.396 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:43:19.864 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:43:19.952 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:43:20.004 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:43:20.034 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:43:20.065 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:43:20.083 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:43:20.274 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:43:20.329 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:43:20.668 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:43:20.727 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:43:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:43:20.918 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:43:20.978 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:43:21.052 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:43:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:43:21.140 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:43:21.220 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:43:21.257 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:43:21.320 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:43:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:43:21.466 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:43:21.514 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:43:21.563 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:43:21.595 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:43:21.624 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:43:21.657 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:43:21.973 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:43:22.003 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:43:22.092 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:43:22.124 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:43:22.167 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:43:22.623 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:43:22.647 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:43:22.718 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:43:22.786 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:43:22.825 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:43:24.941 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:43:24.979 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:43:25.424 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - AppStream 923 kB/s | 19 MB 00:20 2024-03-29T10:43:25.535 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:43:25.574 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:43:25.658 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:43:25.658 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:43:25.658 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:43:25.658 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:25.683 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:43:33.282 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:04 2024-03-29T10:43:34.799 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-29T10:43:35.351 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:43:35.351 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-03-29T10:43:35.351 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-03-29T10:43:35.351 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-03-29T10:43:35.351 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-03-29T10:43:35.351 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-03-29T10:43:35.351 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-03-29T10:43:35.351 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-03-29T10:43:35.351 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:36.232 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:43:36.257 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:43:36.257 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:43:36.257 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:43:36.258 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:43:36.258 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:43:36.258 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:36.259 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:43:36.326 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:43:36.326 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:43:37.525 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:43:37.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:43:37.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:43:37.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:43:37.526 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:43:37.526 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:43:37.526 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:43:37.526 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:43:37.526 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:43:37.526 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:43:37.526 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:43:37.526 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:43:37.527 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:43:37.528 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:43:37.528 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:43:37.528 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:43:37.528 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:43:37.528 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:43:37.528 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:43:37.528 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:43:37.528 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:43:38.688 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:43:38.688 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:38.688 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:43:38.689 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.690 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.691 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:38.691 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:43:38.691 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:43:38.691 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:43:38.691 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:43:38.691 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:43:38.691 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:38.691 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:43:39.021 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test 2024-03-29T10:43:39.572 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:43:39.726 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:43:39.726 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:43:39.726 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:43:39.726 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2024-03-29T10:43:39.727 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:39.728 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 48 M 2024-03-29T10:43:39.728 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 204 M 2024-03-29T10:43:39.728 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:43:39.939 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:43:39.981 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T10:43:40.014 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:43:40.064 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 563 kB/s | 163 kB 00:00 2024-03-29T10:43:40.140 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T10:43:42.523 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 17 MB/s | 47 MB 00:02 2024-03-29T10:43:42.526 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:42.526 INFO:teuthology.orchestra.run.smithi177.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-29T10:43:42.541 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:43:42.562 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:43:42.563 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:43:42.952 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:43:42.952 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:43:43.537 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:43:43.615 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:43:43.657 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:43:43.707 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:43:43.749 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:43:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:43:48.256 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:43:48.334 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:43:50.721 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:43:50.721 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:43:50.721 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:43:50.722 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:43:50.722 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:43:50.722 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:51.134 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:43:51.418 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph 2024-03-29T10:43:51.998 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:43:52.161 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:43:52.164 INFO:teuthology.orchestra.run.smithi177.stdout:========================================================================================= 2024-03-29T10:43:52.164 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:43:52.164 INFO:teuthology.orchestra.run.smithi177.stdout:========================================================================================= 2024-03-29T10:43:52.164 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:43:52.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:43:52.164 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-29T10:43:52.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:43:52.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:43:52.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:43:52.164 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:43:52.165 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:43:52.166 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:43:52.167 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:43:52.167 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:43:52.167 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:52.167 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:43:52.167 INFO:teuthology.orchestra.run.smithi177.stdout:========================================================================================= 2024-03-29T10:43:52.167 INFO:teuthology.orchestra.run.smithi177.stdout:Install 36 Packages 2024-03-29T10:43:52.167 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:43:52.168 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 30 M 2024-03-29T10:43:52.168 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 107 M 2024-03-29T10:43:52.168 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:43:52.675 INFO:teuthology.orchestra.run.smithi177.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-29T10:43:53.084 INFO:teuthology.orchestra.run.smithi177.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 1.5 MB 00:00 2024-03-29T10:43:53.509 INFO:teuthology.orchestra.run.smithi177.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 5.6 MB/s | 4.7 MB 00:00 2024-03-29T10:43:53.559 INFO:teuthology.orchestra.run.smithi177.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.8 MB/s | 247 kB 00:00 2024-03-29T10:43:53.601 INFO:teuthology.orchestra.run.smithi177.stdout:(5/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 2.1 MB 00:01 2024-03-29T10:43:53.735 INFO:teuthology.orchestra.run.smithi177.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-29T10:43:53.760 INFO:teuthology.orchestra.run.smithi177.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 910 kB/s | 144 kB 00:00 2024-03-29T10:43:53.777 INFO:teuthology.orchestra.run.smithi177.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 755 kB/s | 32 kB 00:00 2024-03-29T10:43:53.793 INFO:teuthology.orchestra.run.smithi177.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:43:53.810 INFO:teuthology.orchestra.run.smithi177.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:43:53.844 INFO:teuthology.orchestra.run.smithi177.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T10:43:53.936 INFO:teuthology.orchestra.run.smithi177.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 9.1 MB/s | 1.1 MB 00:00 2024-03-29T10:43:53.953 INFO:teuthology.orchestra.run.smithi177.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 952 kB/s | 103 kB 00:00 2024-03-29T10:43:53.978 INFO:teuthology.orchestra.run.smithi177.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T10:43:53.995 INFO:teuthology.orchestra.run.smithi177.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T10:43:54.229 INFO:teuthology.orchestra.run.smithi177.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 95 kB/s | 22 kB 00:00 2024-03-29T10:43:54.396 INFO:teuthology.orchestra.run.smithi177.stdout:(17/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 13 MB/s | 17 MB 00:01 2024-03-29T10:43:54.571 INFO:teuthology.orchestra.run.smithi177.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 211 kB/s | 125 kB 00:00 2024-03-29T10:43:54.746 INFO:teuthology.orchestra.run.smithi177.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-03-29T10:43:54.821 INFO:teuthology.orchestra.run.smithi177.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 255 kB/s | 151 kB 00:00 2024-03-29T10:43:55.013 INFO:teuthology.orchestra.run.smithi177.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 56 kB/s | 11 kB 00:00 2024-03-29T10:43:55.072 INFO:teuthology.orchestra.run.smithi177.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 343 kB/s | 172 kB 00:00 2024-03-29T10:43:55.280 INFO:teuthology.orchestra.run.smithi177.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 66 kB/s | 18 kB 00:00 2024-03-29T10:43:55.331 INFO:teuthology.orchestra.run.smithi177.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 614 kB/s | 358 kB 00:00 2024-03-29T10:43:55.356 INFO:teuthology.orchestra.run.smithi177.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 81 kB/s | 23 kB 00:00 2024-03-29T10:43:55.548 INFO:teuthology.orchestra.run.smithi177.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 73 kB/s | 19 kB 00:00 2024-03-29T10:43:55.582 INFO:teuthology.orchestra.run.smithi177.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 78 kB/s | 20 kB 00:00 2024-03-29T10:43:55.699 INFO:teuthology.orchestra.run.smithi177.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 135 kB/s | 46 kB 00:00 2024-03-29T10:43:55.890 INFO:teuthology.orchestra.run.smithi177.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 230 kB/s | 79 kB 00:00 2024-03-29T10:43:55.957 INFO:teuthology.orchestra.run.smithi177.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 63 kB/s | 16 kB 00:00 2024-03-29T10:43:56.099 INFO:teuthology.orchestra.run.smithi177.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 526 kB/s | 272 kB 00:00 2024-03-29T10:43:56.238 INFO:teuthology.orchestra.run.smithi177.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 127 kB/s | 36 kB 00:00 2024-03-29T10:43:56.266 INFO:teuthology.orchestra.run.smithi177.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 239 kB/s | 90 kB 00:00 2024-03-29T10:43:56.537 INFO:teuthology.orchestra.run.smithi177.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 74 kB/s | 20 kB 00:00 2024-03-29T10:43:56.609 INFO:teuthology.orchestra.run.smithi177.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 451 kB/s | 230 kB 00:00 2024-03-29T10:43:56.825 INFO:teuthology.orchestra.run.smithi177.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 728 kB/s | 427 kB 00:00 2024-03-29T10:43:56.833 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:56.834 INFO:teuthology.orchestra.run.smithi177.stdout:Total 6.4 MB/s | 30 MB 00:04 2024-03-29T10:43:56.857 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:43:56.900 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:43:56.900 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:43:57.292 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:43:57.293 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:43:58.596 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:43:58.653 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:43:58.694 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:43:58.722 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:43:58.783 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:43:58.829 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:43:58.861 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:43:58.953 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:43:59.030 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:43:59.073 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:43:59.147 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:43:59.200 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:43:59.238 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:43:59.287 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:43:59.343 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:43:59.525 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:43:59.567 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:43:59.606 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:43:59.640 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:43:59.670 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:43:59.747 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:43:59.781 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:43:59.837 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:43:59.879 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:43:59.914 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:44:00.080 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:44:00.080 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:44:00.081 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:44:00.153 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:44:00.287 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:44:00.413 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:44:00.571 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:44:00.601 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:44:00.627 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:44:00.627 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:00.627 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:44:00.627 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:44:00.628 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:44:00.628 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:02.052 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:44:02.078 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:44:02.079 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:02.079 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:44:02.079 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:44:02.079 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:44:02.079 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:02.519 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:44:02.542 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:44:02.543 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:02.543 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:44:02.543 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:44:02.543 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:44:02.543 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:02.745 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:44:02.768 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:44:02.768 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:02.768 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:44:02.768 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:44:02.768 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:44:02.768 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:02.825 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:44:04.654 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:44:04.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:44:04.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:44:04.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:44:04.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:44:04.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:44:04.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:44:04.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:44:04.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:44:04.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:44:04.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:44:04.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:44:05.589 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:44:05.589 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:05.589 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:05.589 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:44:05.590 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:44:05.591 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:44:05.592 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:44:05.592 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:44:05.592 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:44:05.592 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:44:05.592 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:44:05.592 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:44:05.592 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:44:05.592 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:44:05.592 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:05.592 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:05.941 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base 2024-03-29T10:44:06.510 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:06.609 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:44:06.661 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:06.662 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-29T10:44:06.662 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:06.730 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install cephadm 2024-03-29T10:44:07.292 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:07.442 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:07.443 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:07.443 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:44:07.443 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:07.443 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:44:07.443 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:44:07.444 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:07.444 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:44:07.444 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:07.444 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-29T10:44:07.444 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:07.444 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 747 k 2024-03-29T10:44:07.444 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 754 k 2024-03-29T10:44:07.444 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:44:07.742 INFO:teuthology.orchestra.run.smithi177.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.4 MB/s | 747 kB 00:00 2024-03-29T10:44:07.743 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:07.743 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.4 MB/s | 747 kB 00:00 2024-03-29T10:44:07.743 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:44:07.748 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:44:07.748 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:44:07.753 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:44:07.754 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:44:07.824 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:44:08.234 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:08.329 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:08.706 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:08.974 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:44:08.974 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:08.974 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:08.975 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:08.975 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:08.975 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:09.204 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:44:09.774 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:09.930 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:09.930 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:09.930 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-03-29T10:44:09.930 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:09.930 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:44:09.930 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:44:09.931 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:09.931 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:44:09.931 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:09.931 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-29T10:44:09.931 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:09.931 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 145 k 2024-03-29T10:44:09.931 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 436 k 2024-03-29T10:44:09.931 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:44:10.356 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 341 kB/s | 145 kB 00:00 2024-03-29T10:44:10.356 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:10.356 INFO:teuthology.orchestra.run.smithi177.stdout:Total 339 kB/s | 145 kB 00:00 2024-03-29T10:44:10.357 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:44:10.364 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:44:10.364 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:44:10.417 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:44:10.417 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:44:10.553 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:44:10.632 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:44:10.656 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:44:10.656 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:10.656 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:44:10.656 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:11.197 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 585 kB/s | 21 MB 00:36 2024-03-29T10:44:11.624 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:44:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:11.625 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:11.625 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:11.625 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:11.880 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr 2024-03-29T10:44:12.454 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:12.553 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:44:12.605 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:12.606 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-29T10:44:12.606 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:12.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:44:13.232 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:13.382 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:13.383 INFO:teuthology.orchestra.run.smithi177.stdout:====================================================================================== 2024-03-29T10:44:13.383 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:44:13.383 INFO:teuthology.orchestra.run.smithi177.stdout:====================================================================================== 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:13.384 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:44:13.385 INFO:teuthology.orchestra.run.smithi177.stdout:====================================================================================== 2024-03-29T10:44:13.385 INFO:teuthology.orchestra.run.smithi177.stdout:Install 12 Packages 2024-03-29T10:44:13.385 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:13.385 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 6.9 M 2024-03-29T10:44:13.385 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 26 M 2024-03-29T10:44:13.385 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:44:13.960 INFO:teuthology.orchestra.run.smithi177.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 61 kB/s | 15 kB 00:00 2024-03-29T10:44:13.976 INFO:teuthology.orchestra.run.smithi177.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 118 kB/s | 31 kB 00:00 2024-03-29T10:44:14.126 INFO:teuthology.orchestra.run.smithi177.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.3 MB/s | 1.8 MB 00:00 2024-03-29T10:44:14.243 INFO:teuthology.orchestra.run.smithi177.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-03-29T10:44:14.269 INFO:teuthology.orchestra.run.smithi177.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 815 kB/s | 238 kB 00:00 2024-03-29T10:44:14.627 INFO:teuthology.orchestra.run.smithi177.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 54 kB/s | 19 kB 00:00 2024-03-29T10:44:14.919 INFO:teuthology.orchestra.run.smithi177.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.1 MB/s | 863 kB 00:00 2024-03-29T10:44:14.953 INFO:teuthology.orchestra.run.smithi177.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 773 kB/s | 548 kB 00:00 2024-03-29T10:44:15.061 INFO:teuthology.orchestra.run.smithi177.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-29T10:44:15.187 INFO:teuthology.orchestra.run.smithi177.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-29T10:44:15.245 INFO:teuthology.orchestra.run.smithi177.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-29T10:44:15.462 INFO:teuthology.orchestra.run.smithi177.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 266 kB/s | 144 kB 00:00 2024-03-29T10:44:15.467 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:15.467 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.3 MB/s | 6.9 MB 00:02 2024-03-29T10:44:15.486 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:44:15.503 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:44:15.503 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:44:15.678 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:44:15.678 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:44:16.187 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:44:16.279 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:44:16.314 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:44:16.360 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:44:16.399 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:44:16.439 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-29T10:44:16.499 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:44:16.625 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:44:16.719 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:44:16.762 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:44:16.791 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:44:16.822 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:44:17.065 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:44:17.098 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:44:17.846 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:44:17.846 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:44:17.846 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:44:17.847 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:44:17.847 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:44:17.847 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:44:17.847 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:44:17.847 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:44:17.847 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:44:17.847 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:44:17.847 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:44:17.847 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:44:18.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:44:18.197 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:18.197 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:18.197 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:44:18.197 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:18.197 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:18.197 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:18.198 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:44:18.198 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:44:18.198 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:44:18.198 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:44:18.198 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:44:18.198 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:44:18.198 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:44:18.198 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:44:18.198 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:18.198 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:18.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:44:19.043 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:19.124 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:44:19.132 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:44:19.132 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:44:19.132 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrading: 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout: boost-program-options 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:44:19.133 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:44:19.134 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:44:19.134 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:44:19.134 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:44:19.134 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:44:19.134 INFO:teuthology.orchestra.run.smithi005.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:44:19.134 INFO:teuthology.orchestra.run.smithi005.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:44:19.134 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:44:19.134 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:44:19.137 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:44:19.137 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:44:19.137 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:44:19.137 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:44:19.138 INFO:teuthology.orchestra.run.smithi005.stdout:Installing weak dependencies: 2024-03-29T10:44:19.139 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:44:19.139 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:19.139 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:44:19.139 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:44:19.139 INFO:teuthology.orchestra.run.smithi005.stdout:Install 30 Packages 2024-03-29T10:44:19.139 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrade 2 Packages 2024-03-29T10:44:19.139 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:19.139 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 58 M 2024-03-29T10:44:19.140 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:44:19.195 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:19.196 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-29T10:44:19.196 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:44:19.197 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:44:19.198 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:44:19.198 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:44:19.198 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:19.198 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:44:19.198 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-29T10:44:19.198 INFO:teuthology.orchestra.run.smithi177.stdout:Install 12 Packages 2024-03-29T10:44:19.198 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:19.198 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 37 M 2024-03-29T10:44:19.199 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 209 M 2024-03-29T10:44:19.199 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:44:19.484 INFO:teuthology.orchestra.run.smithi177.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 196 kB/s | 32 kB 00:00 2024-03-29T10:44:19.518 INFO:teuthology.orchestra.run.smithi177.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 517 kB/s | 17 kB 00:00 2024-03-29T10:44:19.560 INFO:teuthology.orchestra.run.smithi177.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 825 kB/s | 34 kB 00:00 2024-03-29T10:44:19.719 INFO:teuthology.orchestra.run.smithi177.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-29T10:44:19.852 INFO:teuthology.orchestra.run.smithi177.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-29T10:44:19.903 INFO:teuthology.orchestra.run.smithi177.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-29T10:44:19.928 INFO:teuthology.orchestra.run.smithi177.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 217 kB 00:00 2024-03-29T10:44:20.148 INFO:teuthology.orchestra.run.smithi005.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 6.4 MB/s | 5.3 MB 00:00 2024-03-29T10:44:20.206 INFO:teuthology.orchestra.run.smithi005.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 432 kB/s | 25 kB 00:00 2024-03-29T10:44:20.220 INFO:teuthology.orchestra.run.smithi177.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-03-29T10:44:20.348 INFO:teuthology.orchestra.run.smithi005.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.1 MB/s | 742 kB 00:00 2024-03-29T10:44:20.379 INFO:teuthology.orchestra.run.smithi177.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-29T10:44:20.437 INFO:teuthology.orchestra.run.smithi177.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 906 kB/s | 195 kB 00:00 2024-03-29T10:44:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.3 MB/s | 498 kB 00:00 2024-03-29T10:44:20.529 INFO:teuthology.orchestra.run.smithi177.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-29T10:44:20.749 INFO:teuthology.orchestra.run.smithi005.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 6.8 MB/s | 9.7 MB 00:01 2024-03-29T10:44:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 774 kB/s | 45 kB 00:00 2024-03-29T10:44:20.884 INFO:teuthology.orchestra.run.smithi005.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-29T10:44:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:(8/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.0 MB/s | 5.1 MB 00:00 2024-03-29T10:44:21.033 INFO:teuthology.orchestra.run.smithi005.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.1 MB/s | 162 kB 00:00 2024-03-29T10:44:21.076 INFO:teuthology.orchestra.run.smithi005.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 4.7 MB/s | 320 kB 00:00 2024-03-29T10:44:21.109 INFO:teuthology.orchestra.run.smithi005.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-29T10:44:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T10:44:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:(13/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 9.8 MB/s | 20 MB 00:02 2024-03-29T10:44:21.460 INFO:teuthology.orchestra.run.smithi005.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 302 kB/s | 106 kB 00:00 2024-03-29T10:44:21.485 INFO:teuthology.orchestra.run.smithi005.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 454 kB/s | 159 kB 00:00 2024-03-29T10:44:21.502 INFO:teuthology.orchestra.run.smithi005.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 354 kB/s | 44 kB 00:00 2024-03-29T10:44:21.661 INFO:teuthology.orchestra.run.smithi005.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 661 kB 00:00 2024-03-29T10:44:21.686 INFO:teuthology.orchestra.run.smithi005.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 282 kB 00:00 2024-03-29T10:44:21.703 INFO:teuthology.orchestra.run.smithi005.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 205 kB/s | 41 kB 00:00 2024-03-29T10:44:21.728 INFO:teuthology.orchestra.run.smithi005.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T10:44:21.745 INFO:teuthology.orchestra.run.smithi005.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 596 kB/s | 35 kB 00:00 2024-03-29T10:44:21.762 INFO:teuthology.orchestra.run.smithi005.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 793 kB/s | 46 kB 00:00 2024-03-29T10:44:22.021 INFO:teuthology.orchestra.run.smithi005.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 100 kB/s | 26 kB 00:00 2024-03-29T10:44:22.271 INFO:teuthology.orchestra.run.smithi005.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 588 kB/s | 309 kB 00:00 2024-03-29T10:44:22.363 INFO:teuthology.orchestra.run.smithi005.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-29T10:44:22.629 INFO:teuthology.orchestra.run.smithi005.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 188 kB/s | 67 kB 00:00 2024-03-29T10:44:22.855 INFO:teuthology.orchestra.run.smithi005.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-03-29T10:44:23.055 INFO:teuthology.orchestra.run.smithi005.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-29T10:44:23.080 INFO:teuthology.orchestra.run.smithi005.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 425 kB/s | 192 kB 00:00 2024-03-29T10:44:23.220 INFO:teuthology.orchestra.run.smithi177.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.9 MB/s | 7.4 MB 00:03 2024-03-29T10:44:23.224 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:23.225 INFO:teuthology.orchestra.run.smithi177.stdout:Total 9.2 MB/s | 37 MB 00:04 2024-03-29T10:44:23.269 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:44:23.294 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:44:23.294 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:44:23.439 INFO:teuthology.orchestra.run.smithi005.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 8.7 MB/s | 3.1 MB 00:00 2024-03-29T10:44:23.497 INFO:teuthology.orchestra.run.smithi005.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 7.6 MB/s | 3.4 MB 00:00 2024-03-29T10:44:23.646 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:44:23.646 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:44:23.650 INFO:teuthology.orchestra.run.smithi005.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-03-29T10:44:23.654 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:23.654 INFO:teuthology.orchestra.run.smithi005.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-29T10:44:23.655 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:44:23.697 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:44:23.697 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:44:24.205 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:44:24.205 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:44:24.359 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:44:24.435 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:44:24.466 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:44:24.561 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:44:24.602 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:44:24.780 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:44:24.810 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:44:24.887 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:44:25.543 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:44:25.656 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:44:25.699 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:44:25.779 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:44:25.810 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:44:25.836 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:44:25.867 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:44:25.883 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:44:26.043 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:44:26.099 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:44:26.492 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:44:26.521 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:44:26.596 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:44:26.686 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:44:26.886 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:44:26.895 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:44:26.968 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:44:26.989 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:44:27.024 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:44:27.094 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:44:27.116 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:44:27.163 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:44:27.199 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:44:27.309 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:44:27.341 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:44:27.381 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:44:27.414 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:44:27.443 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:44:27.476 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:44:27.800 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:44:27.830 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:44:27.931 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:44:27.938 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:44:27.959 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:44:27.960 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:44:28.002 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:44:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:44:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:44:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:44:28.658 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:44:28.693 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:44:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:44:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:44:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:44:30.426 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:44:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:44:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:44:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:44:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:44:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:44:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:44:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:44:30.427 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:44:30.879 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:44:30.906 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:44:30.906 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:30.906 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:30.906 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:30.906 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:30.907 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:30.915 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:44:31.288 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:44:31.490 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:44:31.527 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:44:31.615 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:44:31.616 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:44:31.616 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:44:31.616 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:31.653 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:44:31.863 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:32.019 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:32.020 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-03-29T10:44:32.020 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:44:32.020 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-03-29T10:44:32.020 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:44:32.020 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:44:32.020 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-29T10:44:32.020 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout:Install 13 Packages 2024-03-29T10:44:32.021 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:32.022 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.0 M 2024-03-29T10:44:32.022 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 26 M 2024-03-29T10:44:32.022 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:44:32.243 INFO:teuthology.orchestra.run.smithi177.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 170 kB/s | 24 kB 00:00 2024-03-29T10:44:32.268 INFO:teuthology.orchestra.run.smithi177.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 18 kB 00:00 2024-03-29T10:44:32.302 INFO:teuthology.orchestra.run.smithi177.stdout:(3/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 259 kB/s | 52 kB 00:00 2024-03-29T10:44:32.327 INFO:teuthology.orchestra.run.smithi177.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-29T10:44:32.352 INFO:teuthology.orchestra.run.smithi177.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-29T10:44:32.378 INFO:teuthology.orchestra.run.smithi177.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-29T10:44:32.403 INFO:teuthology.orchestra.run.smithi177.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 568 kB/s | 43 kB 00:00 2024-03-29T10:44:32.653 INFO:teuthology.orchestra.run.smithi177.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-29T10:44:32.678 INFO:teuthology.orchestra.run.smithi177.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 99 kB/s | 32 kB 00:00 2024-03-29T10:44:32.904 INFO:teuthology.orchestra.run.smithi177.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 262 kB/s | 59 kB 00:00 2024-03-29T10:44:32.929 INFO:teuthology.orchestra.run.smithi177.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 386 kB/s | 203 kB 00:00 2024-03-29T10:44:33.079 INFO:teuthology.orchestra.run.smithi177.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-03-29T10:44:33.230 INFO:teuthology.orchestra.run.smithi177.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-29T10:44:33.231 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:33.232 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-29T10:44:33.232 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:44:33.247 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:44:33.247 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:44:33.345 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:44:33.345 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:44:33.865 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:44:34.033 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:44:34.082 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:44:34.181 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:44:34.223 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:44:34.316 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:44:34.416 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:44:34.482 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:44:35.197 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:44:35.246 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:44:35.280 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:44:35.328 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:44:35.350 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:44:36.139 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:44:36.139 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:44:36.139 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:44:36.139 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:44:36.139 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:44:36.139 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:44:36.139 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:44:36.140 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:44:36.140 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:44:36.140 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:44:36.140 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:44:36.140 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:44:36.140 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:44:36.502 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:44:36.503 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:44:36.504 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:44:36.504 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:36.504 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:36.782 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:44:37.347 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:37.499 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:37.500 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-03-29T10:44:37.500 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:44:37.500 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-03-29T10:44:37.500 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:44:37.500 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:44:37.501 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-29T10:44:37.501 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:44:37.501 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:44:37.501 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:44:37.501 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:44:37.502 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:44:37.502 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:37.502 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:44:37.502 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-03-29T10:44:37.502 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2024-03-29T10:44:37.503 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:37.504 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 6.8 M 2024-03-29T10:44:37.504 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 33 M 2024-03-29T10:44:37.504 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:44:38.206 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T10:44:38.231 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 443 kB/s | 138 kB 00:00 2024-03-29T10:44:38.298 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 655 kB/s | 248 kB 00:00 2024-03-29T10:44:38.682 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 133 kB/s | 51 kB 00:00 2024-03-29T10:44:41.539 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:44:41.539 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /sys 2024-03-29T10:44:41.539 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /proc 2024-03-29T10:44:41.539 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /mnt 2024-03-29T10:44:41.539 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /var/tmp 2024-03-29T10:44:41.540 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /home 2024-03-29T10:44:41.540 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /root 2024-03-29T10:44:41.540 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /tmp 2024-03-29T10:44:41.540 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:41.632 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 17 kB/s | 58 kB 00:03 2024-03-29T10:44:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:44:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:44:42.489 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:42.490 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:44:42.490 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:44:42.490 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:44:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:44:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:44:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:44:43.648 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:44:43.649 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:44:43.650 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:44:43.650 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:44:43.650 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:44:43.650 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:44:43.651 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:44:43.651 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:44:43.651 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:44:43.651 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:44:43.651 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:44:44.821 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout:Upgraded: 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:44:44.822 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:44:44.823 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.824 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.824 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:44.824 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:44:44.824 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:44:44.824 INFO:teuthology.orchestra.run.smithi005.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:44:44.824 INFO:teuthology.orchestra.run.smithi005.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:44:44.824 INFO:teuthology.orchestra.run.smithi005.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:44:44.824 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:44.824 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:44:45.082 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-test 2024-03-29T10:44:45.606 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 74 kB/s | 548 kB 00:07 2024-03-29T10:44:45.609 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:45.610 INFO:teuthology.orchestra.run.smithi177.stdout:Total 858 kB/s | 6.8 MB 00:08 2024-03-29T10:44:45.625 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:44:45.636 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:44:45.636 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:44:45.647 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:44:45.730 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:44:45.730 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:44:45.803 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:44:45.804 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:44:45.804 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:44:45.804 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:44:45.804 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:44:45.804 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:44:45.804 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-29T10:44:45.804 INFO:teuthology.orchestra.run.smithi005.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:44:45.804 INFO:teuthology.orchestra.run.smithi005.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:44:45.804 INFO:teuthology.orchestra.run.smithi005.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:44:45.805 INFO:teuthology.orchestra.run.smithi005.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:44:45.805 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:44:45.805 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:45.805 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:44:45.805 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:44:45.805 INFO:teuthology.orchestra.run.smithi005.stdout:Install 6 Packages 2024-03-29T10:44:45.805 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:45.805 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 48 M 2024-03-29T10:44:45.806 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 204 M 2024-03-29T10:44:45.806 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:44:46.025 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:44:46.186 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:44:46.232 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:44:46.241 INFO:teuthology.orchestra.run.smithi005.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:44:46.283 INFO:teuthology.orchestra.run.smithi005.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-29T10:44:46.316 INFO:teuthology.orchestra.run.smithi005.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:44:46.450 INFO:teuthology.orchestra.run.smithi005.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 433 kB/s | 163 kB 00:00 2024-03-29T10:44:46.534 INFO:teuthology.orchestra.run.smithi005.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 863 kB/s | 187 kB 00:00 2024-03-29T10:44:46.744 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:44:46.826 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:44:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:44:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:44:47.685 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:44:47.685 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:44:47.685 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:44:47.685 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:44:47.685 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:44:47.685 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:44:47.983 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:44:47.983 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:47.984 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:47.984 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:47.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:44:47.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:44:47.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:44:47.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:44:47.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:44:47.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:47.984 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:48.228 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse 2024-03-29T10:44:48.425 INFO:teuthology.orchestra.run.smithi005.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-29T10:44:48.429 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:48.429 INFO:teuthology.orchestra.run.smithi005.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-29T10:44:48.445 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:44:48.468 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:44:48.468 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:44:48.791 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:48.867 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:44:48.867 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:44:48.941 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:48.942 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:48.942 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:44:48.942 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:48.942 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:44:48.942 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:44:48.942 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-29T10:44:48.942 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:44:48.942 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:48.942 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:44:48.942 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:48.943 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2024-03-29T10:44:48.943 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:48.943 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 964 k 2024-03-29T10:44:48.943 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.8 M 2024-03-29T10:44:48.943 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:44:49.157 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 368 kB/s | 79 kB 00:00 2024-03-29T10:44:49.463 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:44:49.543 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:44:49.585 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:44:49.625 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:44:49.657 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 885 kB 00:00 2024-03-29T10:44:49.658 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:49.659 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.3 MB/s | 964 kB 00:00 2024-03-29T10:44:49.659 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:44:49.659 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:44:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:44:49.679 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:44:49.679 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:44:49.753 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:44:49.753 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:44:49.940 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:44:50.083 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:44:50.172 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:44:50.913 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:44:50.913 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:44:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:44:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:51.250 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:44:51.250 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:51.250 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:51.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-volume 2024-03-29T10:44:52.111 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:52.265 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:52.265 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:52.265 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-29T10:44:52.265 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:52.265 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:44:52.265 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:44:52.265 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-29T10:44:52.266 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:44:52.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:52.266 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:44:52.266 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:52.266 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2024-03-29T10:44:52.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:52.266 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 332 k 2024-03-29T10:44:52.267 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.5 M 2024-03-29T10:44:52.267 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:44:52.386 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 575 kB/s | 69 kB 00:00 2024-03-29T10:44:52.511 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.1 MB/s | 264 kB 00:00 2024-03-29T10:44:52.513 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:52.513 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-29T10:44:52.513 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:44:52.521 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:44:52.521 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:44:52.561 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:44:52.562 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:44:52.711 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:44:52.888 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:44:52.931 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:44:52.947 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:44:52.947 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:44:52.947 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:44:52.947 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:53.675 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:44:54.040 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:44:54.040 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:54.040 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:54.041 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:44:54.041 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:44:54.041 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:54.041 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:54.359 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel 2024-03-29T10:44:54.417 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:44:54.497 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:44:54.933 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:55.083 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:55.084 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:55.084 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-03-29T10:44:55.084 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:55.084 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:44:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:44:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:44:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-29T10:44:55.085 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 127 k 2024-03-29T10:44:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 456 k 2024-03-29T10:44:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:44:55.426 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 372 kB/s | 127 kB 00:00 2024-03-29T10:44:55.426 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:55.427 INFO:teuthology.orchestra.run.smithi177.stdout:Total 370 kB/s | 127 kB 00:00 2024-03-29T10:44:55.427 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:44:55.433 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:44:55.433 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:44:55.488 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:44:55.489 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:44:55.635 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:44:55.732 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:44:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:44:56.473 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:44:56.473 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:56.473 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:56.474 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:56.711 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2 2024-03-29T10:44:56.855 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:44:56.855 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:44:56.856 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:44:56.856 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:44:56.856 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:44:56.856 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:44:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:44:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:57.233 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:44:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:44:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:44:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:44:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:44:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:57.234 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:44:57.283 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:57.385 INFO:teuthology.orchestra.run.smithi177.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:44:57.439 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:57.440 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-29T10:44:57.440 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:57.513 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs-devel 2024-03-29T10:44:57.544 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph 2024-03-29T10:44:58.085 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:44:58.128 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:44:58.237 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:44:58.238 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:58.238 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-03-29T10:44:58.238 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:58.238 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:44:58.239 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:44:58.239 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:58.239 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:44:58.239 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:44:58.239 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-29T10:44:58.239 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:58.239 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 31 k 2024-03-29T10:44:58.239 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 140 k 2024-03-29T10:44:58.239 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:44:58.291 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:44:58.293 INFO:teuthology.orchestra.run.smithi005.stdout:========================================================================================= 2024-03-29T10:44:58.293 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:44:58.293 INFO:teuthology.orchestra.run.smithi005.stdout:========================================================================================= 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:44:58.294 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:44:58.295 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout:========================================================================================= 2024-03-29T10:44:58.296 INFO:teuthology.orchestra.run.smithi005.stdout:Install 36 Packages 2024-03-29T10:44:58.297 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:44:58.297 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 30 M 2024-03-29T10:44:58.297 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 107 M 2024-03-29T10:44:58.297 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:44:58.412 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 180 kB/s | 31 kB 00:00 2024-03-29T10:44:58.412 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:58.412 INFO:teuthology.orchestra.run.smithi177.stdout:Total 179 kB/s | 31 kB 00:00 2024-03-29T10:44:58.412 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:44:58.416 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:44:58.416 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:44:58.451 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:44:58.452 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:44:58.580 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:44:58.678 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:44:58.811 INFO:teuthology.orchestra.run.smithi005.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-03-29T10:44:58.953 INFO:teuthology.orchestra.run.smithi005.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-03-29T10:44:59.070 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:44:59.245 INFO:teuthology.orchestra.run.smithi005.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.3 MB/s | 2.1 MB 00:00 2024-03-29T10:44:59.337 INFO:teuthology.orchestra.run.smithi005.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.6 MB/s | 247 kB 00:00 2024-03-29T10:44:59.338 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:44:59.339 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:59.339 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:44:59.339 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:44:59.339 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:44:59.339 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:44:59.456 INFO:teuthology.orchestra.run.smithi005.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T10:44:59.495 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2 2024-03-29T10:44:59.498 INFO:teuthology.orchestra.run.smithi005.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:44:59.532 INFO:teuthology.orchestra.run.smithi005.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 943 kB/s | 32 kB 00:00 2024-03-29T10:44:59.566 INFO:teuthology.orchestra.run.smithi005.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:44:59.641 INFO:teuthology.orchestra.run.smithi005.stdout:(9/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 5.6 MB/s | 4.7 MB 00:00 2024-03-29T10:44:59.666 INFO:teuthology.orchestra.run.smithi005.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 386 kB/s | 39 kB 00:00 2024-03-29T10:44:59.692 INFO:teuthology.orchestra.run.smithi005.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T10:44:59.742 INFO:teuthology.orchestra.run.smithi005.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:44:59.767 INFO:teuthology.orchestra.run.smithi005.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T10:44:59.792 INFO:teuthology.orchestra.run.smithi005.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:44:59.817 INFO:teuthology.orchestra.run.smithi005.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-29T10:44:59.984 INFO:teuthology.orchestra.run.smithi005.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 16 MB/s | 17 MB 00:01 2024-03-29T10:45:00.026 INFO:teuthology.orchestra.run.smithi005.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 107 kB/s | 22 kB 00:00 2024-03-29T10:45:00.081 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:45:00.184 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:00.238 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:45:00.239 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-29T10:45:00.240 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:45:00.260 INFO:teuthology.orchestra.run.smithi005.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 186 kB/s | 43 kB 00:00 2024-03-29T10:45:00.291 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1 2024-03-29T10:45:00.318 INFO:teuthology.orchestra.run.smithi005.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 452 kB/s | 151 kB 00:00 2024-03-29T10:45:00.477 INFO:teuthology.orchestra.run.smithi005.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 796 kB/s | 172 kB 00:00 2024-03-29T10:45:00.569 INFO:teuthology.orchestra.run.smithi005.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-03-29T10:45:00.644 INFO:teuthology.orchestra.run.smithi005.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-29T10:45:00.736 INFO:teuthology.orchestra.run.smithi005.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-03-29T10:45:00.762 INFO:teuthology.orchestra.run.smithi005.stdout:(24/36): python3-requests-2.25.1-8.el9.noarch.r 129 kB/s | 125 kB 00:00 2024-03-29T10:45:00.787 INFO:teuthology.orchestra.run.smithi005.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 766 kB/s | 358 kB 00:00 2024-03-29T10:45:00.812 INFO:teuthology.orchestra.run.smithi005.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 260 kB/s | 19 kB 00:00 2024-03-29T10:45:00.854 INFO:teuthology.orchestra.run.smithi005.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-29T10:45:00.878 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:45:00.896 INFO:teuthology.orchestra.run.smithi005.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 943 kB/s | 79 kB 00:00 2024-03-29T10:45:00.977 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:01.013 INFO:teuthology.orchestra.run.smithi005.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 205 kB/s | 46 kB 00:00 2024-03-29T10:45:01.028 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:45:01.029 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-29T10:45:01.029 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:45:01.046 INFO:teuthology.orchestra.run.smithi005.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 109 kB/s | 16 kB 00:00 2024-03-29T10:45:01.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados 2024-03-29T10:45:01.080 INFO:teuthology.orchestra.run.smithi005.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-03-29T10:45:01.114 INFO:teuthology.orchestra.run.smithi005.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-03-29T10:45:01.139 INFO:teuthology.orchestra.run.smithi005.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 714 kB/s | 90 kB 00:00 2024-03-29T10:45:01.215 INFO:teuthology.orchestra.run.smithi005.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-03-29T10:45:01.265 INFO:teuthology.orchestra.run.smithi005.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-29T10:45:01.348 INFO:teuthology.orchestra.run.smithi005.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-03-29T10:45:01.352 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:01.352 INFO:teuthology.orchestra.run.smithi005.stdout:Total 9.7 MB/s | 30 MB 00:03 2024-03-29T10:45:01.377 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:45:01.423 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:45:01.423 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:45:01.646 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:45:01.746 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:01.798 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:45:01.799 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-29T10:45:01.799 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:45:01.836 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:45:01.836 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:45:01.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw 2024-03-29T10:45:02.415 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:45:02.513 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:02.565 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:45:02.566 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-29T10:45:02.566 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:45:02.612 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs 2024-03-29T10:45:03.180 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:45:03.279 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:45:03.330 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:45:03.331 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-29T10:45:03.331 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:45:03.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd 2024-03-29T10:45:03.393 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:45:03.430 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:45:03.462 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:45:03.536 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:45:03.700 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:45:03.736 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:45:03.833 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:45:03.908 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:45:03.955 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:45:03.955 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:45:04.030 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:45:04.054 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:04.082 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:45:04.106 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:45:04.107 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-29T10:45:04.107 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:45:04.119 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:45:04.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse 2024-03-29T10:45:04.178 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:45:04.234 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:45:04.430 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:45:04.474 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:45:04.512 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:45:04.555 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:45:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:45:04.658 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:45:04.717 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:45:04.820 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:45:04.854 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:45:04.867 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:45:04.868 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:45:04.868 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-03-29T10:45:04.868 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:45:04.868 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:45:04.868 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:45:04.868 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:04.868 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:45:04.869 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:45:04.869 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-29T10:45:04.869 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:04.869 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 85 k 2024-03-29T10:45:04.869 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 231 k 2024-03-29T10:45:04.869 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:45:04.895 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:45:04.938 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:45:04.972 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:45:05.068 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:45:05.103 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:45:05.180 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:45:05.181 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 273 kB/s | 85 kB 00:00 2024-03-29T10:45:05.182 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:05.182 INFO:teuthology.orchestra.run.smithi177.stdout:Total 272 kB/s | 85 kB 00:00 2024-03-29T10:45:05.183 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:45:05.189 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:45:05.189 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:45:05.245 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:45:05.245 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:45:05.338 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:45:05.380 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:45:05.471 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:45:05.480 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:45:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:45:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:45:05.691 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:05.691 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:45:05.691 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:45:05.691 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:45:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:05.868 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:06.145 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:06.145 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:06.145 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:45:06.146 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:06.146 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:06.146 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:45:06.356 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror 2024-03-29T10:45:06.931 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:45:07.089 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:45:07.089 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 3.1 M 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 13 M 2024-03-29T10:45:07.090 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:45:07.200 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:45:07.226 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:45:07.226 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:07.227 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:45:07.227 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:45:07.227 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:45:07.227 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:07.634 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.7 MB/s | 3.1 MB 00:00 2024-03-29T10:45:07.634 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:07.635 INFO:teuthology.orchestra.run.smithi177.stdout:Total 5.7 MB/s | 3.1 MB 00:00 2024-03-29T10:45:07.635 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:45:07.644 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:45:07.644 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:45:07.692 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:45:07.715 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:45:07.716 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:07.716 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:45:07.716 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:45:07.716 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:45:07.716 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:07.716 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:45:07.717 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:45:07.935 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:45:07.958 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:45:07.958 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:07.958 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:45:07.958 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:45:07.958 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:45:07.958 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:08.018 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:45:08.144 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:45:08.201 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:08.226 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:08.227 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:08.227 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:45:08.227 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:45:08.227 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:45:08.227 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:09.371 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:09.371 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:09.371 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:45:09.371 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:09.371 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:09.371 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:45:09.596 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd 2024-03-29T10:45:09.928 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:45:09.928 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:45:09.928 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:45:09.928 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:45:09.928 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:45:09.928 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:45:09.928 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:45:09.928 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:45:09.929 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:45:09.930 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:45:09.930 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:45:09.930 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:45:09.930 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:45:09.930 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:45:09.930 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:45:09.930 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:45:09.931 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:45:09.931 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:45:09.931 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:45:09.931 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:45:09.931 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:45:09.931 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:45:09.931 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:45:09.932 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:45:10.169 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:02:00 ago on Fri 29 Mar 2024 10:43:10 AM UTC. 2024-03-29T10:45:10.317 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-29T10:45:10.317 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 171 k 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 494 k 2024-03-29T10:45:10.318 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-29T10:45:10.673 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 484 kB/s | 171 kB 00:00 2024-03-29T10:45:10.673 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:10.674 INFO:teuthology.orchestra.run.smithi177.stdout:Total 482 kB/s | 171 kB 00:00 2024-03-29T10:45:10.674 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-29T10:45:10.682 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-29T10:45:10.682 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-29T10:45:10.738 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-29T10:45:10.738 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-29T10:45:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-29T10:45:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:45:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:10.969 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:45:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:45:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:45:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:45:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:45:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:45:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:45:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:45:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:10.972 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:45:10.990 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:11.285 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-base 2024-03-29T10:45:11.433 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:11.729 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:45:11.729 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:11.729 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-29T10:45:11.729 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:11.729 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:45:11.729 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-29T10:45:11.857 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:45:11.959 INFO:teuthology.orchestra.run.smithi005.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:11.968 DEBUG:teuthology.parallel:result is None 2024-03-29T10:45:12.014 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:45:12.015 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-03-29T10:45:12.015 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:45:12.090 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install cephadm 2024-03-29T10:45:12.662 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:45:12.817 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:45:12.818 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:45:12.818 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:45:12.818 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:45:12.818 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:45:12.818 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:45:12.818 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:12.819 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:45:12.819 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:45:12.819 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-03-29T10:45:12.819 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:12.819 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 747 k 2024-03-29T10:45:12.819 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 754 k 2024-03-29T10:45:12.819 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:45:13.263 INFO:teuthology.orchestra.run.smithi005.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.6 MB/s | 747 kB 00:00 2024-03-29T10:45:13.263 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:13.263 INFO:teuthology.orchestra.run.smithi005.stdout:Total 1.6 MB/s | 747 kB 00:00 2024-03-29T10:45:13.263 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:45:13.269 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:45:13.269 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:45:13.275 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:45:13.275 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:45:13.351 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:45:13.795 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:45:13.878 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:45:14.276 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:45:14.548 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:45:14.548 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:14.548 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:45:14.548 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:45:14.549 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:14.549 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:45:14.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:45:15.356 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:45:15.514 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:45:15.515 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:45:15.515 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-03-29T10:45:15.515 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:45:15.516 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:45:15.516 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:45:15.516 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:15.516 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:45:15.516 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:45:15.516 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-03-29T10:45:15.516 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:15.516 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 145 k 2024-03-29T10:45:15.516 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 436 k 2024-03-29T10:45:15.516 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:45:15.853 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 432 kB/s | 145 kB 00:00 2024-03-29T10:45:15.854 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:15.854 INFO:teuthology.orchestra.run.smithi005.stdout:Total 430 kB/s | 145 kB 00:00 2024-03-29T10:45:15.854 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:45:15.862 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:45:15.862 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:45:15.917 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:45:15.917 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:45:16.059 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:45:16.120 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:45:16.145 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:45:16.145 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:16.145 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:45:16.145 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:45:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:17.089 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:45:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:17.089 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:17.089 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:45:17.344 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr 2024-03-29T10:45:17.920 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:45:18.022 INFO:teuthology.orchestra.run.smithi005.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:45:18.078 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:45:18.079 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-03-29T10:45:18.079 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:45:18.159 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:45:18.728 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:45:18.887 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:45:18.888 INFO:teuthology.orchestra.run.smithi005.stdout:====================================================================================== 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout:====================================================================================== 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:45:18.889 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stdout:====================================================================================== 2024-03-29T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stdout:Install 12 Packages 2024-03-29T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:18.891 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 6.9 M 2024-03-29T10:45:18.891 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 26 M 2024-03-29T10:45:18.891 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:45:19.482 INFO:teuthology.orchestra.run.smithi005.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 79 kB/s | 15 kB 00:00 2024-03-29T10:45:19.582 INFO:teuthology.orchestra.run.smithi005.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 106 kB/s | 31 kB 00:00 2024-03-29T10:45:19.650 INFO:teuthology.orchestra.run.smithi005.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.9 MB/s | 1.8 MB 00:00 2024-03-29T10:45:19.683 INFO:teuthology.orchestra.run.smithi005.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T10:45:19.708 INFO:teuthology.orchestra.run.smithi005.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-29T10:45:19.992 INFO:teuthology.orchestra.run.smithi005.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 69 kB/s | 19 kB 00:00 2024-03-29T10:45:20.084 INFO:teuthology.orchestra.run.smithi005.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.9 MB/s | 863 kB 00:00 2024-03-29T10:45:20.209 INFO:teuthology.orchestra.run.smithi005.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T10:45:20.293 INFO:teuthology.orchestra.run.smithi005.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 374 kB/s | 31 kB 00:00 2024-03-29T10:45:20.376 INFO:teuthology.orchestra.run.smithi005.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T10:45:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.7 MB/s | 2.0 MB 00:00 2024-03-29T10:45:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 376 kB/s | 144 kB 00:00 2024-03-29T10:45:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:Total 4.3 MB/s | 6.9 MB 00:01 2024-03-29T10:45:20.492 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:45:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:45:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:45:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:45:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:45:21.204 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:45:21.278 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:45:21.313 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:45:21.361 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:45:21.390 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:45:21.490 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:45:21.621 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:45:21.713 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:45:21.771 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:45:21.798 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:45:21.830 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:45:22.086 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:45:22.121 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:45:22.986 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:45:22.987 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:45:23.448 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:45:23.449 INFO:teuthology.orchestra.run.smithi005.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:45:23.449 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:45:23.449 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:45:23.449 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:23.449 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:45:23.705 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:45:24.277 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:45:24.436 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:45:24.437 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================================ 2024-03-29T10:45:24.437 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:45:24.437 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================================ 2024-03-29T10:45:24.437 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:45:24.437 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:45:24.438 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:45:24.439 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:45:24.439 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:24.439 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:45:24.439 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================================ 2024-03-29T10:45:24.439 INFO:teuthology.orchestra.run.smithi005.stdout:Install 12 Packages 2024-03-29T10:45:24.439 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:24.439 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 37 M 2024-03-29T10:45:24.440 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 209 M 2024-03-29T10:45:24.440 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:45:24.728 INFO:teuthology.orchestra.run.smithi005.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 246 kB/s | 32 kB 00:00 2024-03-29T10:45:24.751 INFO:teuthology.orchestra.run.smithi005.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 727 kB/s | 17 kB 00:00 2024-03-29T10:45:24.785 INFO:teuthology.orchestra.run.smithi005.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T10:45:24.835 INFO:teuthology.orchestra.run.smithi005.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T10:45:25.010 INFO:teuthology.orchestra.run.smithi005.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-29T10:45:25.069 INFO:teuthology.orchestra.run.smithi005.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T10:45:25.094 INFO:teuthology.orchestra.run.smithi005.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-29T10:45:25.403 INFO:teuthology.orchestra.run.smithi005.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-29T10:45:25.495 INFO:teuthology.orchestra.run.smithi005.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 8.2 MB/s | 7.4 MB 00:00 2024-03-29T10:45:25.745 INFO:teuthology.orchestra.run.smithi005.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 982 kB/s | 245 kB 00:00 2024-03-29T10:45:25.929 INFO:teuthology.orchestra.run.smithi005.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 372 kB/s | 195 kB 00:00 2024-03-29T10:45:25.970 INFO:teuthology.orchestra.run.smithi005.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 919 kB/s | 803 kB 00:00 2024-03-29T10:45:25.973 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:25.973 INFO:teuthology.orchestra.run.smithi005.stdout:Total 24 MB/s | 37 MB 00:01 2024-03-29T10:45:26.017 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:45:26.044 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:45:26.044 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:45:26.426 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:45:26.426 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:45:27.156 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:45:27.236 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:45:27.275 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:45:27.373 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:45:27.412 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:45:27.597 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:45:27.636 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:45:27.721 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:45:28.413 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:45:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:45:29.848 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:45:30.943 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:45:30.977 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:45:33.504 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:45:33.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:45:33.965 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:45:33.966 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:45:33.967 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:45:33.967 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:45:33.967 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:33.967 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:45:34.319 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:45:34.895 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:45:35.052 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:45:35.053 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================== 2024-03-29T10:45:35.053 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:45:35.053 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================== 2024-03-29T10:45:35.053 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:45:35.053 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:45:35.053 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-29T10:45:35.053 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:45:35.053 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================== 2024-03-29T10:45:35.054 INFO:teuthology.orchestra.run.smithi005.stdout:Install 13 Packages 2024-03-29T10:45:35.055 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:45:35.055 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 2.0 M 2024-03-29T10:45:35.055 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 26 M 2024-03-29T10:45:35.055 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:45:35.229 INFO:teuthology.orchestra.run.smithi005.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 198 kB/s | 24 kB 00:00 2024-03-29T10:45:35.254 INFO:teuthology.orchestra.run.smithi005.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 122 kB/s | 18 kB 00:00 2024-03-29T10:45:35.313 INFO:teuthology.orchestra.run.smithi005.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2024-03-29T10:45:35.338 INFO:teuthology.orchestra.run.smithi005.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-29T10:45:35.364 INFO:teuthology.orchestra.run.smithi005.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:45:35.389 INFO:teuthology.orchestra.run.smithi005.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-29T10:45:35.448 INFO:teuthology.orchestra.run.smithi005.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 153 kB/s | 52 kB 00:00 2024-03-29T10:45:36.556 INFO:teuthology.orchestra.run.smithi005.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 27 kB/s | 32 kB 00:01 2024-03-29T10:45:36.798 INFO:teuthology.orchestra.run.smithi005.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 4.2 MB/s | 1.0 MB 00:00 2024-03-29T10:45:36.873 INFO:teuthology.orchestra.run.smithi005.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 789 kB/s | 59 kB 00:00 2024-03-29T10:45:36.940 INFO:teuthology.orchestra.run.smithi005.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-29T10:45:38.615 INFO:teuthology.orchestra.run.smithi005.stdout:(12/13): python3-certifi-2021.10.8-2.el9.noarch 4.6 kB/s | 15 kB 00:03 2024-03-29T10:46:39.475 INFO:teuthology.orchestra.run.smithi005.stdout:(13/13): python3-google-auth-2.28.2-1.el9.noarc 3.2 kB/s | 203 kB 01:04 2024-03-29T10:46:39.477 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:39.477 INFO:teuthology.orchestra.run.smithi005.stdout:Total 33 kB/s | 2.0 MB 01:04 2024-03-29T10:46:39.477 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:46:39.494 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:46:39.494 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:46:39.596 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:46:39.597 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:46:40.121 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:46:40.295 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:46:40.343 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:46:40.399 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:46:40.435 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:46:40.477 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:46:40.574 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:46:40.682 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:46:40.729 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:46:41.486 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:46:41.533 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:46:41.567 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:46:41.616 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:46:41.645 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:46:42.372 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:46:42.373 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:46:42.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:46:42.728 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:46:42.729 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:42.730 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:46:43.003 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:46:43.583 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:46:43.741 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:46:43.741 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================== 2024-03-29T10:46:43.741 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:46:43.741 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================== 2024-03-29T10:46:43.742 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:46:43.742 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:46:43.742 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-29T10:46:43.742 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:46:43.742 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:46:43.742 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:46:43.742 INFO:teuthology.orchestra.run.smithi005.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:46:43.742 INFO:teuthology.orchestra.run.smithi005.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:46:43.742 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:43.743 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:46:43.743 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================== 2024-03-29T10:46:43.743 INFO:teuthology.orchestra.run.smithi005.stdout:Install 6 Packages 2024-03-29T10:46:43.743 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:43.744 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 6.8 M 2024-03-29T10:46:43.744 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 33 M 2024-03-29T10:46:43.744 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:46:44.243 INFO:teuthology.orchestra.run.smithi005.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T10:46:44.344 INFO:teuthology.orchestra.run.smithi005.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 375 kB/s | 138 kB 00:00 2024-03-29T10:46:44.519 INFO:teuthology.orchestra.run.smithi005.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 332 kB/s | 58 kB 00:00 2024-03-29T10:46:44.546 INFO:teuthology.orchestra.run.smithi005.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-29T10:46:44.569 INFO:teuthology.orchestra.run.smithi005.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-29T10:46:44.662 INFO:teuthology.orchestra.run.smithi005.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 362 kB/s | 248 kB 00:00 2024-03-29T10:46:44.664 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:44.664 INFO:teuthology.orchestra.run.smithi005.stdout:Total 7.4 MB/s | 6.8 MB 00:00 2024-03-29T10:46:44.677 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:46:44.691 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:46:44.691 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:46:44.779 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:46:44.785 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:46:45.083 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:46:45.252 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:46:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:46:45.875 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:46:45.962 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:46:46.029 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:46:46.059 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:46:46.882 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:46:46.882 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:46:46.882 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:46:46.882 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:46:46.882 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:46:46.882 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:46:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:46:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:47.177 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:46:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:46:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:46:47.177 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:46:47.178 INFO:teuthology.orchestra.run.smithi005.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:46:47.178 INFO:teuthology.orchestra.run.smithi005.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:46:47.178 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:47.178 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:46:47.409 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-fuse 2024-03-29T10:46:47.977 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:46:48.133 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:46:48.134 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:48.134 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:46:48.134 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:48.134 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:46:48.134 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:46:48.134 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-29T10:46:48.134 INFO:teuthology.orchestra.run.smithi005.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:46:48.134 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:48.135 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:46:48.135 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:48.135 INFO:teuthology.orchestra.run.smithi005.stdout:Install 2 Packages 2024-03-29T10:46:48.135 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:48.135 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 964 k 2024-03-29T10:46:48.135 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 2.8 M 2024-03-29T10:46:48.135 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:46:48.263 INFO:teuthology.orchestra.run.smithi005.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 618 kB/s | 79 kB 00:00 2024-03-29T10:46:48.580 INFO:teuthology.orchestra.run.smithi005.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 885 kB 00:00 2024-03-29T10:46:48.581 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:48.581 INFO:teuthology.orchestra.run.smithi005.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-29T10:46:48.582 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:46:48.603 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:46:48.604 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:46:48.681 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:46:48.681 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:46:48.869 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:46:49.011 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:46:49.084 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:46:49.814 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:46:49.814 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:46:50.126 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:46:50.126 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:50.126 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:46:50.126 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:46:50.126 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:50.126 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:46:50.426 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-volume 2024-03-29T10:46:51.005 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:46:51.163 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:46:51.163 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:51.163 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-29T10:46:51.164 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:51.164 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:46:51.164 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:46:51.164 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-29T10:46:51.164 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:46:51.164 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:51.164 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:46:51.164 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:51.164 INFO:teuthology.orchestra.run.smithi005.stdout:Install 2 Packages 2024-03-29T10:46:51.164 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:51.165 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 332 k 2024-03-29T10:46:51.165 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 1.5 M 2024-03-29T10:46:51.165 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:46:51.294 INFO:teuthology.orchestra.run.smithi005.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 533 kB/s | 69 kB 00:00 2024-03-29T10:46:51.444 INFO:teuthology.orchestra.run.smithi005.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 946 kB/s | 264 kB 00:00 2024-03-29T10:46:51.445 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:51.445 INFO:teuthology.orchestra.run.smithi005.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T10:46:51.446 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:46:51.455 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:46:51.456 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:46:51.492 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:46:51.492 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:46:51.640 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:46:51.824 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:46:51.863 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:46:51.879 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:46:51.879 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:51.879 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:46:51.879 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:46:52.918 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:46:52.918 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:52.918 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:46:52.918 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:52.918 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:46:52.918 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:52.918 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:46:53.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librados-devel 2024-03-29T10:46:53.811 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:37 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:46:53.968 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:53.969 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 127 k 2024-03-29T10:46:53.970 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 456 k 2024-03-29T10:46:53.970 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:46:54.459 INFO:teuthology.orchestra.run.smithi005.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 259 kB/s | 127 kB 00:00 2024-03-29T10:46:54.459 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:54.459 INFO:teuthology.orchestra.run.smithi005.stdout:Total 258 kB/s | 127 kB 00:00 2024-03-29T10:46:54.459 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:46:54.465 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:46:54.465 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:46:54.524 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:46:54.524 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:46:54.678 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:46:54.776 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:55.467 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:55.468 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:46:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:55.468 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:46:55.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install libcephfs2 2024-03-29T10:46:56.293 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:46:56.394 INFO:teuthology.orchestra.run.smithi005.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:56.451 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:46:56.452 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-03-29T10:46:56.452 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:46:56.534 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install libcephfs-devel 2024-03-29T10:46:57.110 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:46:57.268 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:46:57.268 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 31 k 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 140 k 2024-03-29T10:46:57.269 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:46:57.705 INFO:teuthology.orchestra.run.smithi005.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 71 kB/s | 31 kB 00:00 2024-03-29T10:46:57.706 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:57.706 INFO:teuthology.orchestra.run.smithi005.stdout:Total 71 kB/s | 31 kB 00:00 2024-03-29T10:46:57.706 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:46:57.710 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:46:57.710 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:46:57.740 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:46:57.740 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:46:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:46:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:58.347 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:58.616 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:46:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:46:58.617 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:46:58.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librados2 2024-03-29T10:46:59.356 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:43 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:46:59.455 INFO:teuthology.orchestra.run.smithi005.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:59.510 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:46:59.511 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-03-29T10:46:59.511 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:46:59.567 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librbd1 2024-03-29T10:47:00.141 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:44 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:47:00.241 INFO:teuthology.orchestra.run.smithi005.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:47:00.295 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:47:00.296 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-03-29T10:47:00.296 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:47:00.350 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rados 2024-03-29T10:47:00.927 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:44 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:47:01.028 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:47:01.083 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:47:01.084 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-03-29T10:47:01.084 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:47:01.132 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rgw 2024-03-29T10:47:01.710 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:45 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:47:01.811 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:47:01.866 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:47:01.867 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-03-29T10:47:01.867 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:47:01.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-cephfs 2024-03-29T10:47:02.504 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:46 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:47:02.606 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:47:02.661 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:47:02.662 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-03-29T10:47:02.662 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:47:02.717 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rbd 2024-03-29T10:47:03.294 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:47 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:47:03.406 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:47:03.462 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:47:03.463 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-03-29T10:47:03.463 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:47:03.512 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-fuse 2024-03-29T10:47:04.084 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:48 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:47:04.239 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:47:04.240 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:47:04.240 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-03-29T10:47:04.240 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:47:04.240 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:47:04.241 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:47:04.241 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:04.241 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:47:04.241 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:47:04.241 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-03-29T10:47:04.241 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:04.241 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 85 k 2024-03-29T10:47:04.241 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 231 k 2024-03-29T10:47:04.241 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:47:04.559 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 268 kB/s | 85 kB 00:00 2024-03-29T10:47:04.560 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:04.560 INFO:teuthology.orchestra.run.smithi005.stdout:Total 267 kB/s | 85 kB 00:00 2024-03-29T10:47:04.560 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:47:04.568 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:47:04.568 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:47:04.625 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:47:04.626 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:47:04.764 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:47:04.852 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:05.218 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:05.501 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:05.501 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:05.501 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:47:05.501 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:47:05.501 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:05.501 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:47:05.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-mirror 2024-03-29T10:47:06.288 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:50 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:47:06.445 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:47:06.446 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:47:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-03-29T10:47:06.446 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:47:06.446 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:47:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:47:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:06.446 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:47:06.446 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:47:06.446 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-03-29T10:47:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:06.447 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 3.1 M 2024-03-29T10:47:06.447 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 13 M 2024-03-29T10:47:06.447 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:47:07.396 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.3 MB/s | 3.1 MB 00:00 2024-03-29T10:47:07.397 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:07.397 INFO:teuthology.orchestra.run.smithi005.stdout:Total 3.2 MB/s | 3.1 MB 00:00 2024-03-29T10:47:07.397 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:47:07.408 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:47:07.408 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:47:07.486 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:47:07.487 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:47:07.942 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:47:07.997 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:08.021 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:08.021 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:47:08.021 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:47:08.021 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:47:08.021 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:47:08.021 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:09.207 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:09.207 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:09.207 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:47:09.207 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:47:09.207 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:09.207 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:47:09.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-nbd 2024-03-29T10:47:10.034 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:54 ago on Fri 29 Mar 2024 10:44:16 AM UTC. 2024-03-29T10:47:10.190 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-29T10:47:10.190 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 171 k 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 494 k 2024-03-29T10:47:10.191 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-29T10:47:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 294 kB/s | 171 kB 00:00 2024-03-29T10:47:10.776 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:10.776 INFO:teuthology.orchestra.run.smithi005.stdout:Total 293 kB/s | 171 kB 00:00 2024-03-29T10:47:10.776 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-29T10:47:10.785 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-29T10:47:10.785 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-29T10:47:10.844 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-29T10:47:10.845 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-29T10:47:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-29T10:47:11.093 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:11.513 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:11.821 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:11.821 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:11.821 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-29T10:47:11.821 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:47:11.821 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:11.822 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-29T10:47:12.046 DEBUG:teuthology.parallel:result is None 2024-03-29T10:47:12.047 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:47:12.266 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:47:12.286 INFO:teuthology.orchestra.run.smithi005.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:47:12.287 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:47:12.287 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:47:12.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:47:12.500 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:47:12.525 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:47:12.526 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:47:12.526 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:47:12.528 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:47:12.529 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:12.529 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:47:12.557 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:12.557 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:47:12.597 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:47:12.598 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:12.598 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:47:12.628 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:47:12.696 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:12.696 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:47:12.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:47:12.796 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:47:12.796 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:12.796 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:47:12.823 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:47:12.890 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:12.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:47:12.919 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:47:12.985 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:47:12.986 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:12.986 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:47:13.012 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:47:13.078 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:13.079 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:47:13.106 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:47:13.173 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:47:13.258 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:47:13.258 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /var/log/ceph 2024-03-29T10:47:13.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2024-03-29T10:47:13.291 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:47:13.291 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:47:13.327 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:47:13.359 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:47:13.359 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:47:13.393 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:47:13.447 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:47:13.447 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:47:13.447 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622656', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi005.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzB2nmx/jDNQL6ad35f2kCeylwaq4F/KlMlDoy0UNN5CCNeOuTbKQ3wP8tds7fvlFZfqRPeNnBzFN4JMvVnTcM=', 'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcegcc0yDoT9QN1F1Kn+v508+jLs+9MnP5xfCQMWLaSXylIex7wd+H/dWc3zhPQGh+3o98lkHn9JWeyby1HAZo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:47:13.448 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:47:13.461 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:47:13.481 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:47:13.516 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:47:13.548 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:13.548 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:47:13.597 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:47:13.597 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 2024-03-29T10:47:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:47:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:13.652 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T10:47:13.652 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:13.652 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:13.652 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-29 10:47:08.314139886 +0000 2024-03-29T10:47:13.652 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-29 10:42:27.243615083 +0000 2024-03-29T10:47:13.653 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-29 10:42:27.243615083 +0000 2024-03-29T10:47:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-29 10:42:27.243615083 +0000 2024-03-29T10:47:13.653 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:47:13.718 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-29T10:47:13.718 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-29T10:47:13.718 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 7.8098e-05 s, 6.6 MB/s 2024-03-29T10:47:13.719 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:47:13.774 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 2024-03-29T10:47:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:47:13.829 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:13.829 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T10:47:13.829 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:13.829 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:13.829 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-29 10:47:08.314139886 +0000 2024-03-29T10:47:13.829 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-29 10:42:27.579607951 +0000 2024-03-29T10:47:13.829 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-29 10:42:27.579607951 +0000 2024-03-29T10:47:13.829 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-29 10:42:27.579607951 +0000 2024-03-29T10:47:13.829 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:47:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-29T10:47:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-29T10:47:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 9.4386e-05 s, 5.4 MB/s 2024-03-29T10:47:13.895 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:47:13.950 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 2024-03-29T10:47:14.004 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:47:14.005 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:14.005 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:47:14.005 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:14.005 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:14.005 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-29 10:47:08.315139865 +0000 2024-03-29T10:47:14.005 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-29 10:42:27.920600713 +0000 2024-03-29T10:47:14.005 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-29 10:42:27.920600713 +0000 2024-03-29T10:47:14.005 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-29 10:42:27.920600713 +0000 2024-03-29T10:47:14.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:47:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-29T10:47:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-29T10:47:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 9.3664e-05 s, 5.5 MB/s 2024-03-29T10:47:14.071 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:47:14.126 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 2024-03-29T10:47:14.180 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:47:14.180 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:14.180 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:47:14.181 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:14.181 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:14.181 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-29 10:47:08.315139865 +0000 2024-03-29T10:47:14.181 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-29 10:42:28.245593814 +0000 2024-03-29T10:47:14.181 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-29 10:42:28.245593814 +0000 2024-03-29T10:47:14.181 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-29 10:42:28.245593814 +0000 2024-03-29T10:47:14.181 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:47:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-29T10:47:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-29T10:47:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 9.921e-05 s, 5.2 MB/s 2024-03-29T10:47:14.246 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:47:14.301 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:14.301 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:14.301 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:47:14.317 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:47:14.318 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-03-29T10:47:14.372 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:47:14.372 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:14.373 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T10:47:14.373 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:14.373 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:14.373 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-29 10:45:08.475964720 +0000 2024-03-29T10:47:14.373 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-29 10:42:25.186601937 +0000 2024-03-29T10:47:14.373 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-29 10:42:25.186601937 +0000 2024-03-29T10:47:14.373 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-29 10:42:25.186601937 +0000 2024-03-29T10:47:14.373 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:47:14.439 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-29T10:47:14.440 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-29T10:47:14.440 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000186603 s, 2.7 MB/s 2024-03-29T10:47:14.441 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:47:14.501 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-03-29T10:47:14.551 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:47:14.551 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:14.551 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T10:47:14.551 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:14.551 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:14.551 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-29 10:45:08.475964720 +0000 2024-03-29T10:47:14.552 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-29 10:42:25.525592669 +0000 2024-03-29T10:47:14.552 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-29 10:42:25.525592669 +0000 2024-03-29T10:47:14.552 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-29 10:42:25.525592669 +0000 2024-03-29T10:47:14.552 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:47:14.617 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-29T10:47:14.617 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-29T10:47:14.617 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000195085 s, 2.6 MB/s 2024-03-29T10:47:14.618 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:47:14.674 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-03-29T10:47:14.729 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:47:14.729 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:14.729 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T10:47:14.729 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:14.729 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:14.729 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-29 10:45:08.476964693 +0000 2024-03-29T10:47:14.729 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-29 10:42:25.864583401 +0000 2024-03-29T10:47:14.729 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-29 10:42:25.864583401 +0000 2024-03-29T10:47:14.729 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-29 10:42:25.864583401 +0000 2024-03-29T10:47:14.730 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:47:14.794 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-29T10:47:14.794 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-29T10:47:14.795 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000161412 s, 3.2 MB/s 2024-03-29T10:47:14.795 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:47:14.851 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-03-29T10:47:14.906 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:47:14.906 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:14.906 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T10:47:14.906 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:14.906 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:14.906 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-29 10:45:08.476964693 +0000 2024-03-29T10:47:14.906 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-29 10:42:26.188574543 +0000 2024-03-29T10:47:14.906 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-29 10:42:26.188574543 +0000 2024-03-29T10:47:14.907 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-29 10:42:26.188574543 +0000 2024-03-29T10:47:14.907 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:47:14.972 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-29T10:47:14.972 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-29T10:47:14.972 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000187402 s, 2.7 MB/s 2024-03-29T10:47:14.973 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:47:15.029 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:15.029 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:47:15.029 INFO:tasks.ceph:Generating config... 2024-03-29T10:47:15.030 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:47:15.030 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:47:15.030 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:47:15.030 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:47:15.031 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:47:15.031 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:47:15.031 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:47:15.031 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:47:15.031 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:47:15.031 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:47:15.031 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:47:15.031 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:47:15.031 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:47:15.031 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:47:15.031 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:47:15.031 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:47:15.031 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:47:15.031 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:47:15.032 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:47:15.032 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:47:15.032 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:47:15.032 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:47:15.032 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:47:15.033 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:47:15.033 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:47:15.034 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:47:15.086 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:47:15.089 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:47:15.135 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:47:15.163 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.5:6789]'), ('mon.c', '[v1:172.21.15.5:6790]'), ('mon.b', '[v1:172.21.15.177:6789]')] 2024-03-29T10:47:15.164 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.5:6789],[v1:172.21.15.5:6790],[v1:172.21.15.177:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:47:15.165 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:15.165 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:47:15.219 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.5:6789]' --addv c '[v1:172.21.15.5:6790]' --addv b '[v1:172.21.15.177:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:15.304 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: generated fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = quincy 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 0 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:min_mon_release 17 (quincy) 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:election_strategy: 1 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:15.305 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:47:15.306 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:47:15.321 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fcc4bb0a-729e-4ad6-a413-fad605d2bb4a... 2024-03-29T10:47:15.321 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:47:15.363 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:47:15.411 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2024-03-29T10:47:15.411 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2024-03-29T10:47:15.411 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:47:15.411 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2024-03-29T10:47:15.411 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.412 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:47:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "[v1:172.21.15.5:6789],[v1:172.21.15.5:6790],[v1:172.21.15.177:6789]" 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: lockdep = True 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: mon client directed command retry = 5 2024-03-29T10:47:15.414 INFO:teuthology.orchestra.run.smithi005.stdout: mon election default strategy = 1 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: ms bind msgr2 = False 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject socket failures = 5000 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: ms type = async 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.415 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug trim objects = true 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:47:15.416 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore compression algorithm = zstd 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore compression mode = aggressive 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore zero block detection = True 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: debug monc = 20 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:47:15.417 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd class default list = * 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd class load list = * 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd client watch timeout = 120 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify cached snaps = True 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify missing on start = True 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock override recovery settings = True 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd op queue = debug_random 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd op queue cut off = debug_random 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: osd shutdown pgref assert = True 2024-03-29T10:47:15.418 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2024-03-29T10:47:15.419 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:47:15.420 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: mon min osdmap epochs = 50 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune interval = 2 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune min = 15 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: mon scrub interval = 300 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on pool no app = False 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: paxos service trim min = 10 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:47:15.421 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:47:15.422 INFO:teuthology.orchestra.run.smithi005.stdout: debug monc = 20 2024-03-29T10:47:15.422 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-03-29T10:47:15.422 INFO:teuthology.orchestra.run.smithi005.stdout: debug objecter = 20 2024-03-29T10:47:15.422 INFO:teuthology.orchestra.run.smithi005.stdout: debug rados = 20 2024-03-29T10:47:15.422 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2024-03-29T10:47:15.422 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2024-03-29T10:47:15.422 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2024-03-29T10:47:15.422 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-03-29T10:47:15.422 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-03-29T10:47:15.422 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:47:15.423 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-03-29T10:47:15.424 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "[v1:172.21.15.5:6789],[v1:172.21.15.5:6790],[v1:172.21.15.177:6789]" 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: lockdep = True 2024-03-29T10:47:15.425 INFO:teuthology.orchestra.run.smithi177.stdout: mon client directed command retry = 5 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: mon election default strategy = 1 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: ms bind msgr2 = False 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject socket failures = 5000 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: ms type = async 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.426 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-03-29T10:47:15.427 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore compression algorithm = zstd 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore compression mode = aggressive 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore zero block detection = True 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 20 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 20 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-29T10:47:15.428 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 10 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: osd class default list = * 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: osd class load list = * 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: osd client watch timeout = 120 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify cached snaps = True 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify missing on start = True 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock override recovery settings = True 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue = debug_random 2024-03-29T10:47:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue cut off = debug_random 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: osd shutdown pgref assert = True 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-29T10:47:15.430 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:47:15.431 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: mon min osdmap epochs = 50 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune interval = 2 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune min = 15 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: mon scrub interval = 300 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on pool no app = False 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: paxos service trim min = 10 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-03-29T10:47:15.432 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-03-29T10:47:15.433 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-03-29T10:47:15.433 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:47:15.433 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:47:15.433 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2024-03-29T10:47:15.433 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-29T10:47:15.433 INFO:teuthology.orchestra.run.smithi177.stdout: debug objecter = 20 2024-03-29T10:47:15.433 INFO:teuthology.orchestra.run.smithi177.stdout: debug rados = 20 2024-03-29T10:47:15.433 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-03-29T10:47:15.433 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-03-29T10:47:15.433 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-03-29T10:47:15.434 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:47:15.435 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:47:15.530 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:47:15.530 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:15.530 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:47:15.544 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:15.544 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:47:15.598 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T10:47:15.599 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:15.599 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:47:15.599 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:47:15.677 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:15.677 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:15.732 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T10:47:15.732 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:15.732 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:47:15.732 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:47:15.773 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:15.773 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:15.828 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:47:15.828 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:47:15.828 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:47:15.882 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:47:15.885 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:47:15.943 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:47:15.945 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:47:15.945 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:47:15.946 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:47:15.987 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:47:16.002 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:47:16.044 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:47:16.059 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:47:16.059 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:47:16.059 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:47:16.085 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:16.085 INFO:tasks.ceph:role: osd.0 2024-03-29T10:47:16.086 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T10:47:16.086 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:47:16.152 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:16.152 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:16.152 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:16.153 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:16.153 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:16.153 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:16.153 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:16.153 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:16.153 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:16.153 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:16.184 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-03-29T10:47:16.185 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-03-29T10:47:16.186 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:47:16.257 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:47:16.328 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:47:16.394 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:16.394 INFO:tasks.ceph:role: osd.1 2024-03-29T10:47:16.394 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T10:47:16.394 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:47:16.460 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:16.460 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:16.460 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:16.460 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:16.460 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:16.460 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:16.460 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:16.460 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:16.460 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:16.461 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:16.492 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-03-29T10:47:16.494 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-03-29T10:47:16.494 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:47:16.564 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:47:16.638 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:47:16.705 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:16.705 INFO:tasks.ceph:role: osd.2 2024-03-29T10:47:16.705 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T10:47:16.705 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:47:16.771 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:16.771 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:16.771 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:16.771 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:16.772 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:16.772 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:16.772 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:16.772 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:16.772 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:16.772 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:16.804 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-03-29T10:47:16.806 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-03-29T10:47:16.806 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:47:16.876 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:47:16.949 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:47:17.015 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:17.015 INFO:tasks.ceph:role: osd.3 2024-03-29T10:47:17.015 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T10:47:17.015 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:47:17.081 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:17.081 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:17.081 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:17.081 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:17.082 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:17.082 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:17.082 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:17.113 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-03-29T10:47:17.115 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-03-29T10:47:17.115 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:47:17.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:47:17.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:17.372+0000 7fa7def8f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:47:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:17.372+0000 7fa7def8f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:47:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:17.372+0000 7fa7def8f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:47:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:17.372+0000 7fa7def8f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:47:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:17.372+0000 7fa7def8f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:47:17.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:17.372+0000 7fa7def8f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:47:20.198 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:47:20.229 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:20.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:20.338+0000 7fd1e6a17640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:47:20.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:20.338+0000 7fd1e6a17640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:47:20.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:20.338+0000 7fd1e6a17640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:47:20.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:20.338+0000 7fd1e6a17640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:47:20.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:20.338+0000 7fd1e6a17640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:47:20.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:20.339+0000 7fd1e6a17640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:47:23.172 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:47:23.203 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:23.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:23.312+0000 7f89c9413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:47:23.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:23.312+0000 7f89c9413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:47:23.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:23.312+0000 7f89c9413640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:47:23.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:23.312+0000 7f89c9413640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:47:23.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:23.312+0000 7f89c9413640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:47:23.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:23.313+0000 7f89c9413640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:47:26.143 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:47:26.174 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:26.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:26.285+0000 7fe08ab9a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:47:26.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:26.285+0000 7fe08ab9a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:47:26.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:26.285+0000 7fe08ab9a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:47:26.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:26.285+0000 7fe08ab9a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:47:26.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:26.285+0000 7fe08ab9a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:47:26.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:26.286+0000 7fe08ab9a640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:47:29.123 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:47:29.154 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:47:29.187 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:29.188 INFO:tasks.ceph:role: osd.4 2024-03-29T10:47:29.188 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T10:47:29.188 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:47:29.265 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:29.266 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:29.266 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:29.266 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:29.266 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:29.266 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:29.266 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:29.266 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:29.266 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:29.266 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:29.784 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-03-29T10:47:29.786 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-03-29T10:47:29.786 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:47:29.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:47:29.949 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:47:30.018 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:30.018 INFO:tasks.ceph:role: osd.5 2024-03-29T10:47:30.019 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T10:47:30.019 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:47:30.095 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:30.095 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:30.095 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:30.095 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:30.095 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:30.095 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:30.095 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:30.095 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:30.096 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:30.096 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:30.546 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-03-29T10:47:30.548 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-03-29T10:47:30.549 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:47:30.643 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:47:30.676 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:47:30.745 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:30.745 INFO:tasks.ceph:role: osd.6 2024-03-29T10:47:30.745 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T10:47:30.745 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:47:30.821 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:30.821 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:30.821 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:30.821 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:30.821 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:30.822 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:30.822 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:30.822 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:30.822 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:30.822 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:31.313 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-03-29T10:47:31.315 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-03-29T10:47:31.316 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:47:31.462 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:47:31.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:47:31.567 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:31.567 INFO:tasks.ceph:role: osd.7 2024-03-29T10:47:31.567 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T10:47:31.567 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:47:31.644 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:31.644 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:31.645 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:31.645 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:31.645 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:31.645 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:32.140 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-03-29T10:47:32.143 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-03-29T10:47:32.143 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:47:32.270 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:47:32.308 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:32.408 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:32.426+0000 7fa63ea0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:47:32.408 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:32.426+0000 7fa63ea0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:47:32.408 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:32.426+0000 7fa63ea0f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:47:32.408 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:32.426+0000 7fa63ea0f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:47:32.409 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:32.426+0000 7fa63ea0f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:47:32.409 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:32.427+0000 7fa63ea0f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:47:35.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:47:35.293 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:35.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:35.401+0000 7fbd4880f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:47:35.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:35.402+0000 7fbd4880f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:47:35.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:35.402+0000 7fbd4880f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:47:35.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:35.402+0000 7fbd4880f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:47:35.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:35.402+0000 7fbd4880f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:47:35.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:35.402+0000 7fbd4880f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:47:38.230 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:47:38.261 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:38.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:38.372+0000 7f3b7f9bb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:47:38.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:38.373+0000 7f3b7f9bb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:47:38.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:38.373+0000 7f3b7f9bb640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:47:38.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:38.373+0000 7f3b7f9bb640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:47:38.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:38.373+0000 7f3b7f9bb640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:47:38.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:38.373+0000 7f3b7f9bb640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:47:41.209 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:47:41.244 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:41.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:41.353+0000 7f41a080e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:47:41.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:41.353+0000 7f41a080e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:47:41.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:41.353+0000 7f41a080e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:47:41.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:41.353+0000 7f41a080e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:47:41.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:41.353+0000 7f41a080e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:47:41.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:41.353+0000 7f41a080e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:47:44.183 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:47:44.214 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:47:44.214 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:44.214 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:47:44.246 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:44.246 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:47:44.317 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:44.317 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:47:44.386 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:44.386 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:47:44.454 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:44.455 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:47:44.523 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:44.523 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:47:44.554 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:44.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:47:44.625 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:44.625 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:47:44.695 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:44.695 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:47:44.765 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:44.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:47:44.833 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:44.833 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:47:44.849 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:44.849 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:47:44.889 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:47:44.889 DEBUG:teuthology.orchestra.run.smithi005:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:47:44.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.y] 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAznAZmt5q+NRAA+t0ONpNafJGL7fLVQxkrvA== 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA1nAZmL6RBFhAAGfG/ALRPG4646vBhq8Fxsg== 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA4nAZmJwY8FBAAOoTiCk66Af7M14SjrUWv4g== 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA7nAZmJgSzEhAAi/KsJPWxLRGRhjxMU1U19A== 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.3] 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA+nAZmGgMVERAACnRQicFkzwiISnJbKquq5w== 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2024-03-29T10:47:44.933 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAznAZmfS9ZORAAvhzh+KJgnQH0AN0z+tL+LA== 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.4] 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBEnAZm6b5wGRAA2ZPtFW+NiLSC6bJK0OHjyQ== 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.5] 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBHnAZmRVUMGBAA1RbPXEbfPUsj7MMXlHnZsw== 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.6] 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBKnAZm5XZPFhAAhbv+Cv53/k8rAbk+6mH5mQ== 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.7] 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBNnAZmi+0hFRAAzL0GU7N1jwoS6Hj5mVucRA== 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA0nAZmRh5eABAAvYLX3ZiZuDhAB77ptRAFzg== 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout:[client.1] 2024-03-29T10:47:44.934 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA0nAZmRvDCAxAAqel6wLB6N5RK31EXuY+6SQ== 2024-03-29T10:47:44.955 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.y] 2024-03-29T10:47:44.955 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAznAZmt5q+NRAA+t0ONpNafJGL7fLVQxkrvA== 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.0] 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQA1nAZmL6RBFhAAGfG/ALRPG4646vBhq8Fxsg== 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.1] 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQA4nAZmJwY8FBAAOoTiCk66Af7M14SjrUWv4g== 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.2] 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQA7nAZmJgSzEhAAi/KsJPWxLRGRhjxMU1U19A== 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.3] 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQA+nAZmGgMVERAACnRQicFkzwiISnJbKquq5w== 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.x] 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAznAZmfS9ZORAAvhzh+KJgnQH0AN0z+tL+LA== 2024-03-29T10:47:44.956 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.4] 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBEnAZm6b5wGRAA2ZPtFW+NiLSC6bJK0OHjyQ== 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.5] 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBHnAZmRVUMGBAA1RbPXEbfPUsj7MMXlHnZsw== 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.6] 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBKnAZm5XZPFhAAhbv+Cv53/k8rAbk+6mH5mQ== 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.7] 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBNnAZmi+0hFRAAzL0GU7N1jwoS6Hj5mVucRA== 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQA0nAZmRh5eABAAvYLX3ZiZuDhAB77ptRAFzg== 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout:[client.1] 2024-03-29T10:47:44.957 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQA0nAZmRvDCAxAAqel6wLB6N5RK31EXuY+6SQ== 2024-03-29T10:47:44.958 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:47:44.978 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:47:45.052 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.054 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.107 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.110 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.162 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.164 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.216 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.219 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.271 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:47:45.273 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:47:45.325 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.327 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.379 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.381 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.433 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.435 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.486 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.489 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:45.540 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:47:45.543 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:47:45.598 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:47:45.601 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:47:45.655 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:47:45.655 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:47:45.682 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:47:45.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:47:45.805 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:47:45.872 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:47:45.969 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:47:45.995 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:47:46.023 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:47:46.120 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:47:46.146 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:46.149 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:46.201 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:47:46.201 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:47:46.201 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:47:46.203 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:47:46.203 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:47:46.204 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:47:46.205 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:47:46.206 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:47:46.206 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:47:46.244 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:47:46.244 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:47:46.244 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:47:46.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:47:46.246 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:47:46.246 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:47:46.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:47:46.248 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:47:46.248 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:47:46.249 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.368+0000 7f4abf6e4640 10 monclient: get_monmap_and_config 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.368+0000 7f4abf6e4640 10 monclient: build_initial_monmap 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.368+0000 7f4abf6e4640 10 monclient: monmap: 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:46.369948+0000 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:46.369948+0000 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:46.351 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:46.352 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.369+0000 7f4abf6e4640 1 Processor -- start 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.369+0000 7f4abf6e4640 1 -- start start 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.369+0000 7f4abf6e4640 10 monclient: init 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abf6e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abf6e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abf6e4640 10 monclient(hunting): picked mon.noname-b con 0x7f4ab80b8240 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abf6e4640 10 monclient(hunting): picked mon.noname-a con 0x7f4ab80b1f60 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abf6e4640 10 monclient(hunting): picked mon.noname-c con 0x7f4ab809c8c0 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abf6e4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab809bd80 con 0x7f4ab80b1f60 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abf6e4640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab809b960 con 0x7f4ab809c8c0 2024-03-29T10:47:46.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abf6e4640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab80b2ec0 con 0x7f4ab80b8240 2024-03-29T10:47:46.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abf6e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:46.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abf6e4640 10 monclient(hunting): authenticate will time out at 879.611384s 2024-03-29T10:47:46.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abcc7f640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4ab80b1f60 0x7f4ab809c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51152/0 (socket says 172.21.15.5:51152) 2024-03-29T10:47:46.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abcc7f640 1 -- 172.21.15.5:0/3503489343 learned_addr learned my addr 172.21.15.5:0/3503489343 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:47:46.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:46.370+0000 7f4abd480640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f4ab80b8240 0x7f4ab813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:50580/0 (socket says 172.21.15.5:50580) 2024-03-29T10:47:49.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 10 monclient(hunting): tick 2024-03-29T10:47:49.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:47:49.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 1 monclient(hunting): continuing hunt 2024-03-29T10:47:49.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:47:49.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 1 -- 172.21.15.5:0/3503489343 >> v1:172.21.15.5:6790/0 conn(0x7f4ab80b8240 legacy=0x7f4ab813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 1 -- 172.21.15.5:0/3503489343 >> v1:172.21.15.177:6789/0 conn(0x7f4ab809c8c0 legacy=0x7f4ab814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 1 -- 172.21.15.5:0/3503489343 >> v1:172.21.15.5:6789/0 conn(0x7f4ab80b1f60 legacy=0x7f4ab809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:49.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa40036e0 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:49.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f4ab80b8240 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:49.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa400aa70 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:49.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 1 -- 172.21.15.5:0/3503489343 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab813fb60 con 0x7f4aa400aa70 2024-03-29T10:47:49.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 1 -- 172.21.15.5:0/3503489343 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa400f760 con 0x7f4aa40036e0 2024-03-29T10:47:49.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 1 -- 172.21.15.5:0/3503489343 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa40109c0 con 0x7f4ab80b8240 2024-03-29T10:47:49.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:49.370+0000 7f4ab6ffd640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:51.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.539+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613712792 0 0) 0x7f4ab813fb60 con 0x7f4aa400aa70 2024-03-29T10:47:51.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.539+0000 7f4ab7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.539+0000 7f4ab7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.539+0000 7f4ab7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.539+0000 7f4ab7fff640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:47:51.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.539+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa0003520 con 0x7f4aa400aa70 2024-03-29T10:47:51.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950679601 0 0) 0x7f4aa40109c0 con 0x7f4ab80b8240 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab813fb60 con 0x7f4ab80b8240 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236495038 0 0) 0x7f4aa400f760 con 0x7f4aa40036e0 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa40109c0 con 0x7f4aa40036e0 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2243181027 0 0) 0x7f4aa0003520 con 0x7f4aa400aa70 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 >> v1:172.21.15.5:6790/0 conn(0x7f4ab80b8240 legacy=0x7f4aa4009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 >> v1:172.21.15.177:6789/0 conn(0x7f4aa40036e0 legacy=0x7f4aa4005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 1 monclient: found mon.noname-a 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa4011c20 con 0x7f4aa400aa70 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4abf6e4640 5 monclient: authenticate success, global_id 4101 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4abf6e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4aa80016d0 con 0x7f4aa400aa70 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:51.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4ab7fff640 10 monclient: dump: 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.540+0000 7f4abf6e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aa80018a0 con 0x7f4aa400aa70 2024-03-29T10:47:51.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4ab7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4ab7fff640 1 -- 172.21.15.5:0/3503489343 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4aa8001c30 con 0x7f4aa400aa70 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4ab7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4ab7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4ab7fff640 10 monclient: dump: 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4abf6e4640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:51.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4abf6e4640 4 set_mon_vals no callback set 2024-03-29T10:47:51.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4abf6e4640 10 monclient: shutdown 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4abf6e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4abf6e4640 1 -- 172.21.15.5:0/3503489343 >> v1:172.21.15.5:6789/0 conn(0x7f4aa400aa70 legacy=0x7f4aa400cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4abf6e4640 1 -- 172.21.15.5:0/3503489343 shutdown_connections 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.541+0000 7f4abf6e4640 1 -- 172.21.15.5:0/3503489343 >> 172.21.15.5:0/3503489343 conn(0x7f4ab8066b00 msgr2=0x7f4ab8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.542+0000 7f4abf6e4640 1 -- 172.21.15.5:0/3503489343 shutdown_connections 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.542+0000 7f4abf6e4640 1 -- 172.21.15.5:0/3503489343 wait complete. 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.543+0000 7f4abf6e4640 10 monclient: build_initial_monmap 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.543+0000 7f4abf6e4640 10 monclient: monmap: 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:51.533 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.543+0000 7f4abf6e4640 1 librados: starting msgr at 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.543+0000 7f4abf6e4640 1 librados: starting objecter 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.543+0000 7f4abf6e4640 1 Processor -- start 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.543+0000 7f4abf6e4640 1 -- start start 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.543+0000 7f4abf6e4640 1 librados: setting wanted keys 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.543+0000 7f4abf6e4640 1 librados: calling monclient init 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.543+0000 7f4abf6e4640 10 monclient: init 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.544+0000 7f4abf6e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.544+0000 7f4abf6e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.544+0000 7f4abf6e4640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa40036e0 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.544+0000 7f4abf6e4640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa400aa70 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.544+0000 7f4abf6e4640 10 monclient(hunting): picked mon.noname-b con 0x7f4ab80b8240 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.544+0000 7f4abf6e4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab8120040 con 0x7f4aa400aa70 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.544+0000 7f4abcc7f640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4aa400aa70 0x7f4ab81d7530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60030/0 (socket says 172.21.15.5:60030) 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.544+0000 7f4abcc7f640 1 -- 172.21.15.5:0/1052912204 learned_addr learned my addr 172.21.15.5:0/1052912204 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.544+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab81d70a0 con 0x7f4ab80b8240 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.544+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab8120290 con 0x7f4aa40036e0 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4abf6e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4abf6e4640 10 monclient(hunting): authenticate will time out at 884.786165s 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 client.?.objecter ms_handle_connect 0x7f4aa400aa70 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 client.?.objecter ms_handle_connect 0x7f4aa40036e0 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 client.?.objecter ms_handle_connect 0x7f4ab80b8240 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199559077 0 0) 0x7f4ab8120040 con 0x7f4aa400aa70 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 monclient(hunting): my global_id is 4110 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa4011310 con 0x7f4aa400aa70 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611039670 0 0) 0x7f4aa4011310 con 0x7f4aa400aa70 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4aa40118c0 con 0x7f4aa400aa70 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4aac001870 con 0x7f4aa400aa70 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.545+0000 7f4ab5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611622831 0 0) 0x7f4ab8120290 con 0x7f4aa40036e0 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa4011310 con 0x7f4aa40036e0 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1251023021 0 0) 0x7f4aa40118c0 con 0x7f4aa400aa70 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 >> v1:172.21.15.5:6790/0 conn(0x7f4aa40036e0 legacy=0x7f4ab80b1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 >> v1:172.21.15.177:6789/0 conn(0x7f4ab80b8240 legacy=0x7f4ab813a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 1 monclient: found mon.a 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab81d7c40 con 0x7f4aa400aa70 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4abf6e4640 5 monclient: authenticate success, global_id 4110 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4abf6e4640 10 monclient: _renew_subs 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4abf6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab813a760 con 0x7f4aa400aa70 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aac0047e0 con 0x7f4aa400aa70 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4aac004af0 con 0x7f4aa400aa70 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab5ffb640 10 monclient: dump: 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:51.539 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.547+0000 7f4ab77fe640 4 set_mon_vals no callback set 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4aac004fe0 con 0x7f4aa400aa70 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4abf6e4640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4abf6e4640 10 monclient: _renew_subs 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4abf6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab80ad900 con 0x7f4aa400aa70 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4abf6e4640 1 librados: init done 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3187953203 0 0) 0x7f4aac005850 con 0x7f4aa400aa70 2024-03-29T10:47:51.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4ab5ffb640 10 client.4110.objecter ms_dispatch 0x7f4ab8000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:47:51.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4ab5ffb640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:47:51.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4ab5ffb640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:47:51.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.548+0000 7f4ab5ffb640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T10:47:51.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.549+0000 7f4abf6e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.549+0000 7f4abf6e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.549+0000 7f4abf6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:47:51.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.549+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab81d6e70 con 0x7f4aa400aa70 2024-03-29T10:47:51.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.555+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4aac005a90 con 0x7f4aa400aa70 2024-03-29T10:47:51.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.555+0000 7f4ab5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.555+0000 7f4ab5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:51.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.625+0000 7f4abf6e4640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:47:51.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.625+0000 7f4abf6e4640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:47:51.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.625+0000 7f4abf6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:47:51.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.625+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4ab804f730 con 0x7f4aa400aa70 2024-03-29T10:47:51.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.627+0000 7f4ab5ffb640 1 -- 172.21.15.5:0/1052912204 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f4aac005c30 con 0x7f4aa400aa70 2024-03-29T10:47:51.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.627+0000 7f4ab5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:47:51.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.627+0000 7f4ab5ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:47:51.610 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2024-03-29T10:47:51.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.628+0000 7f4abf6e4640 10 librados: watch_flush enter 2024-03-29T10:47:51.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.628+0000 7f4abf6e4640 10 librados: watch_flush exit 2024-03-29T10:47:51.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.628+0000 7f4abf6e4640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T10:47:51.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.628+0000 7f4abf6e4640 10 client.4110.objecter successfully canceled tick 2024-03-29T10:47:51.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.628+0000 7f4abf6e4640 10 monclient: shutdown 2024-03-29T10:47:51.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.628+0000 7f4abf6e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:51.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.628+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 >> v1:172.21.15.5:6789/0 conn(0x7f4aa400aa70 legacy=0x7f4ab81d7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.629+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 shutdown_connections 2024-03-29T10:47:51.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.629+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 >> 172.21.15.5:0/1052912204 conn(0x7f4ab8066b00 msgr2=0x7f4ab813cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:51.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.629+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 shutdown_connections 2024-03-29T10:47:51.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.629+0000 7f4abf6e4640 1 -- 172.21.15.5:0/1052912204 wait complete. 2024-03-29T10:47:51.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:51.629+0000 7f4abf6e4640 1 librados: shutdown 2024-03-29T10:47:51.622 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:47:51.622 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:47:51.623 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:51.623 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:47:51.650 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:51.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:47:51.717 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:51.717 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:47:51.783 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-29T10:47:51.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:47:51.851 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:51.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:47:51.878 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:51.878 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:47:51.945 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:51.945 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:47:52.012 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-29T10:47:52.012 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:47:52.014 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:52.031+0000 7f4f87a17040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:47:52.023 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:52.041+0000 7fc1aea42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:47:52.039 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28 0 2024-03-29T10:47:52.071 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:52.088+0000 7f4f87a17040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:47:52.079 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:52.097+0000 7fc1aea42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:47:52.169 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.186+0000 7f216ec63640 10 monclient: get_monmap_and_config 2024-03-29T10:47:52.169 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.186+0000 7f216ec63640 10 monclient: build_initial_monmap 2024-03-29T10:47:52.169 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.186+0000 7f216ec63640 10 monclient: monmap: 2024-03-29T10:47:52.169 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:52.169 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.169 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:52.188062+0000 2024-03-29T10:47:52.170 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:52.188062+0000 2024-03-29T10:47:52.170 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:52.170 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.170 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:52.170 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:52.170 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:52.170 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.181 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.187+0000 7f216ec63640 1 Processor -- start 2024-03-29T10:47:52.181 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.187+0000 7f216ec63640 1 -- start start 2024-03-29T10:47:52.181 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.187+0000 7f216ec63640 10 monclient: init 2024-03-29T10:47:52.181 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:52.181 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:52.181 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 10 monclient(hunting): picked mon.noname-c con 0x7f216809ac40 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:52.181 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 10 monclient(hunting): picked mon.noname-b con 0x7f216809c870 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:52.181 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 10 monclient(hunting): picked mon.noname-a con 0x7f21680b4360 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216809dec0 con 0x7f21680b4360 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216809da30 con 0x7f216809ac40 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680b39c0 con 0x7f216809c870 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 10 monclient(hunting): authenticate will time out at 887.426780s 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216c9ff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f216809ac40 0x7f21680b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54244/0 (socket says 172.21.15.177:54244) 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216c9ff640 1 -- 172.21.15.177:0/3184326259 learned_addr learned my addr 172.21.15.177:0/3184326259 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180154219 0 0) 0x7f216809da30 con 0x7f216809ac40 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:47:52.182 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2158003520 con 0x7f216809ac40 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225582227 0 0) 0x7f21680b39c0 con 0x7f216809c870 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f216809da30 con 0x7f216809c870 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578468798 0 0) 0x7f2160001490 con 0x7f21680b4360 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21680b39c0 con 0x7f21680b4360 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 364077166 0 0) 0x7f2158003520 con 0x7f216809ac40 2024-03-29T10:47:52.183 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 >> v1:172.21.15.5:6790/0 conn(0x7f216809c870 legacy=0x7f216813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 >> v1:172.21.15.5:6789/0 conn(0x7f21680b4360 legacy=0x7f216814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 monclient: found mon.noname-c 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21680b5160 con 0x7f216809ac40 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f215c001610 con 0x7f216809ac40 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: dump: 2024-03-29T10:47:52.184 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f215c001770 con 0x7f216809ac40 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 1 -- 172.21.15.177:0/3184326259 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f215c001bc0 con 0x7f216809ac40 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.185 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f21677fe640 10 monclient: dump: 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:52.186 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.188+0000 7f216ec63640 4 set_mon_vals no callback set 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.189+0000 7f216ec63640 10 monclient: shutdown 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.189+0000 7f216ec63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.189+0000 7f216ec63640 1 -- 172.21.15.177:0/3184326259 >> v1:172.21.15.177:6789/0 conn(0x7f216809ac40 legacy=0x7f21680b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.189+0000 7f216ec63640 1 -- 172.21.15.177:0/3184326259 shutdown_connections 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.189+0000 7f216ec63640 1 -- 172.21.15.177:0/3184326259 >> 172.21.15.177:0/3184326259 conn(0x7f2168066b00 msgr2=0x7f2168066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.189+0000 7f216ec63640 1 -- 172.21.15.177:0/3184326259 shutdown_connections 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.189+0000 7f216ec63640 1 -- 172.21.15.177:0/3184326259 wait complete. 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.190+0000 7f216ec63640 10 monclient: build_initial_monmap 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.190+0000 7f216ec63640 10 monclient: monmap: 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:52.187 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.191+0000 7f216ec63640 1 librados: starting msgr at 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.191+0000 7f216ec63640 1 librados: starting objecter 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.191+0000 7f216ec63640 1 Processor -- start 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.191+0000 7f216ec63640 1 -- start start 2024-03-29T10:47:52.188 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.191+0000 7f216ec63640 1 librados: setting wanted keys 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.191+0000 7f216ec63640 1 librados: calling monclient init 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.191+0000 7f216ec63640 10 monclient: init 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f216ec63640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f216ec63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f216ec63640 10 monclient(hunting): picked mon.noname-a con 0x7f216809ac40 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f216ec63640 10 monclient(hunting): picked mon.noname-b con 0x7f21681d7f90 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f216ec63640 10 monclient(hunting): picked mon.noname-c con 0x7f21680b08b0 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f216ec63640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680b18d0 con 0x7f216809ac40 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f216ec63640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21681d7380 con 0x7f21681d7f90 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f216ec63640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680af7c0 con 0x7f21680b08b0 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f216ec63640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f216ec63640 10 monclient(hunting): authenticate will time out at 887.430958s 2024-03-29T10:47:52.189 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f2167fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f21681d7f90 0x7f21680b01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54250/0 (socket says 172.21.15.177:54250) 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f2167fff640 1 -- 172.21.15.177:0/1227759334 learned_addr learned my addr 172.21.15.177:0/1227759334 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 client.?.objecter ms_handle_connect 0x7f21681d7f90 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 client.?.objecter ms_handle_connect 0x7f216809ac40 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 client.?.objecter ms_handle_connect 0x7f21680b08b0 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735699236 0 0) 0x7f21681d7380 con 0x7f21681d7f90 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:47:52.190 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f214c004eb0 con 0x7f21681d7f90 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089130485 0 0) 0x7f21680b18d0 con 0x7f216809ac40 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21681d7380 con 0x7f216809ac40 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284223042 0 0) 0x7f21680af7c0 con 0x7f21680b08b0 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21680b18d0 con 0x7f21680b08b0 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 56110528 0 0) 0x7f214c004eb0 con 0x7f21681d7f90 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21680af7c0 con 0x7f21681d7f90 2024-03-29T10:47:52.191 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2154001a30 con 0x7f21681d7f90 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): dump: 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4253576023 0 0) 0x7f21681d7380 con 0x7f216809ac40 2024-03-29T10:47:52.193 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f214c004eb0 con 0x7f216809ac40 2024-03-29T10:47:52.193 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f215c001a80 con 0x7f216809ac40 2024-03-29T10:47:52.193 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.193 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:52.193 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.192+0000 7f21657fa640 10 monclient(hunting): dump: 2024-03-29T10:47:52.193 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.193 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.193 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.193 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1598868070 0 0) 0x7f21680b18d0 con 0x7f21680b08b0 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21681d7380 con 0x7f21680b08b0 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f21600018f0 con 0x7f21680b08b0 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient(hunting): dump: 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.194 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4270772004 0 0) 0x7f21680af7c0 con 0x7f21681d7f90 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 >> v1:172.21.15.5:6790/0 conn(0x7f21680b08b0 legacy=0x7f2168139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 >> v1:172.21.15.5:6789/0 conn(0x7f216809ac40 legacy=0x7f21681d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:52.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 monclient: found mon.b 2024-03-29T10:47:52.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:52.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21680acf60 con 0x7f21681d7f90 2024-03-29T10:47:52.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:52.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:52.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:52.197 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f216ec63640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:47:52.197 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f216ec63640 10 monclient: _renew_subs 2024-03-29T10:47:52.197 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f216ec63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.197 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f216ec63640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2168139720 con 0x7f21681d7f90 2024-03-29T10:47:52.197 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f216ec63640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:52.197 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f216ec63640 10 monclient: _renew_subs 2024-03-29T10:47:52.198 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f216ec63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.198 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f216ec63640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2168139ca0 con 0x7f21681d7f90 2024-03-29T10:47:52.198 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f216ec63640 1 librados: init done 2024-03-29T10:47:52.198 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2154001cf0 con 0x7f21681d7f90 2024-03-29T10:47:52.198 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:52.198 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2154003a60 con 0x7f21681d7f90 2024-03-29T10:47:52.198 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.198 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:52.198 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 monclient: dump: 2024-03-29T10:47:52.199 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.199 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.199 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.199 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.199 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.199 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.200 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.200 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.200 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.200 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f2166ffd640 4 set_mon_vals no callback set 2024-03-29T10:47:52.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2154004010 con 0x7f21681d7f90 2024-03-29T10:47:52.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3751807397 0 0) 0x7f2154006650 con 0x7f21681d7f90 2024-03-29T10:47:52.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 10 client.4114.objecter ms_dispatch 0x7f2168000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:47:52.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:47:52.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:47:52.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.193+0000 7f21657fa640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T10:47:52.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.194+0000 7f216ec63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.194+0000 7f216ec63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.194+0000 7f216ec63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.194+0000 7f216ec63640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2148005080 con 0x7f21681d7f90 2024-03-29T10:47:52.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.196+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2154006950 con 0x7f21681d7f90 2024-03-29T10:47:52.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.196+0000 7f21657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.196+0000 7f21657fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:52.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.268+0000 7f216ec63640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28", "id": 0}] 2024-03-29T10:47:52.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.268+0000 7f216ec63640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28", "id": 0}] 2024-03-29T10:47:52.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.268+0000 7f216ec63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.268+0000 7f216ec63640 1 -- 172.21.15.177:0/1227759334 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28", "id": 0} v 0) v1 -- 0x7f21480058b0 con 0x7f21681d7f90 2024-03-29T10:47:52.255 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.273+0000 7f21657fa640 1 -- 172.21.15.177:0/1227759334 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 337981410 0 732707515) 0x7f2154006b70 con 0x7f21681d7f90 2024-03-29T10:47:52.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.273+0000 7f21657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28", "id": 0}] 2024-03-29T10:47:52.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.273+0000 7f21657fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:52.256 INFO:teuthology.orchestra.run.smithi177.stdout:0 2024-03-29T10:47:52.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 10 librados: watch_flush enter 2024-03-29T10:47:52.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 10 librados: watch_flush exit 2024-03-29T10:47:52.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T10:47:52.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 10 client.4114.objecter successfully canceled tick 2024-03-29T10:47:52.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 10 monclient: shutdown 2024-03-29T10:47:52.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 1 -- 172.21.15.177:0/1227759334 >> v1:172.21.15.177:6789/0 conn(0x7f21681d7f90 legacy=0x7f21680b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 1 -- 172.21.15.177:0/1227759334 shutdown_connections 2024-03-29T10:47:52.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 1 -- 172.21.15.177:0/1227759334 >> 172.21.15.177:0/1227759334 conn(0x7f2168066b00 msgr2=0x7f21681420b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 1 -- 172.21.15.177:0/1227759334 shutdown_connections 2024-03-29T10:47:52.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 1 -- 172.21.15.177:0/1227759334 wait complete. 2024-03-29T10:47:52.258 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.274+0000 7f216ec63640 1 librados: shutdown 2024-03-29T10:47:52.267 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 79b602b0-e74b-4fa1-8418-09e0e3a40335 1 2024-03-29T10:47:52.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.384+0000 7fe65c5ab640 10 monclient: get_monmap_and_config 2024-03-29T10:47:52.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.384+0000 7fe65c5ab640 10 monclient: build_initial_monmap 2024-03-29T10:47:52.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.384+0000 7fe65c5ab640 10 monclient: monmap: 2024-03-29T10:47:52.367 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:52.368 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.368 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:52.385681+0000 2024-03-29T10:47:52.368 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:52.385681+0000 2024-03-29T10:47:52.368 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:52.368 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.368 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:52.368 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:52.368 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:52.368 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.384+0000 7fe65c5ab640 1 Processor -- start 2024-03-29T10:47:52.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.384+0000 7fe65c5ab640 1 -- start start 2024-03-29T10:47:52.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.384+0000 7fe65c5ab640 10 monclient: init 2024-03-29T10:47:52.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65c5ab640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:52.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65c5ab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:52.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65c5ab640 10 monclient(hunting): picked mon.noname-b con 0x7fe65409ac40 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:52.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65c5ab640 10 monclient(hunting): picked mon.noname-a con 0x7fe65409c870 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:52.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65c5ab640 10 monclient(hunting): picked mon.noname-c con 0x7fe6540b4360 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:52.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65c5ab640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65409dec0 con 0x7fe65409c870 2024-03-29T10:47:52.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65c5ab640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65409da30 con 0x7fe6540b4360 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65c5ab640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6540b39c0 con 0x7fe65409ac40 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65c5ab640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65c5ab640 10 monclient(hunting): authenticate will time out at 887.624170s 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65ab48640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fe6540b4360 0x7fe65414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54254/0 (socket says 172.21.15.177:54254) 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe65ab48640 1 -- 172.21.15.177:0/1340305993 learned_addr learned my addr 172.21.15.177:0/1340305993 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804385549 0 0) 0x7fe65409da30 con 0x7fe6540b4360 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe659345640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe659345640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe659345640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe659345640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:47:52.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.385+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe640003520 con 0x7fe6540b4360 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374863986 0 0) 0x7fe644001110 con 0x7fe65409c870 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe65409da30 con 0x7fe65409c870 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31385843 0 0) 0x7fe6540b39c0 con 0x7fe65409ac40 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe644001110 con 0x7fe65409ac40 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2611591228 0 0) 0x7fe640003520 con 0x7fe6540b4360 2024-03-29T10:47:52.382 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 >> v1:172.21.15.5:6790/0 conn(0x7fe65409ac40 legacy=0x7fe6540b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 >> v1:172.21.15.5:6789/0 conn(0x7fe65409c870 legacy=0x7fe65413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: _finish_hunting 0 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 monclient: found mon.noname-c 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540b5160 con 0x7fe6540b4360 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: _finish_auth 0 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: _check_auth_tickets 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe64c001a10 con 0x7fe6540b4360 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: dump: 2024-03-29T10:47:52.383 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe64c001bf0 con 0x7fe6540b4360 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 1 -- 172.21.15.177:0/1340305993 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe64c002000 con 0x7fe6540b4360 2024-03-29T10:47:52.384 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 4 set_mon_vals no callback set 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe659345640 10 monclient: dump: 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 10 monclient: shutdown 2024-03-29T10:47:52.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 1 -- 172.21.15.177:0/1340305993 >> v1:172.21.15.177:6789/0 conn(0x7fe6540b4360 legacy=0x7fe65414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 1 -- 172.21.15.177:0/1340305993 shutdown_connections 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 1 -- 172.21.15.177:0/1340305993 >> 172.21.15.177:0/1340305993 conn(0x7fe654066b00 msgr2=0x7fe654066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 1 -- 172.21.15.177:0/1340305993 shutdown_connections 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.386+0000 7fe65c5ab640 1 -- 172.21.15.177:0/1340305993 wait complete. 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.387+0000 7fe65c5ab640 10 monclient: build_initial_monmap 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.387+0000 7fe65c5ab640 10 monclient: monmap: 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.386 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.387+0000 7fe65c5ab640 1 librados: starting msgr at 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.387+0000 7fe65c5ab640 1 librados: starting objecter 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.387+0000 7fe65c5ab640 1 Processor -- start 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.387+0000 7fe65c5ab640 1 -- start start 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.387+0000 7fe65c5ab640 1 librados: setting wanted keys 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.387+0000 7fe65c5ab640 1 librados: calling monclient init 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.387+0000 7fe65c5ab640 10 monclient: init 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe65c5ab640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe65c5ab640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe65c5ab640 10 monclient(hunting): picked mon.noname-c con 0x7fe65409ac40 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe65c5ab640 10 monclient(hunting): picked mon.noname-b con 0x7fe6541d7cd0 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe65c5ab640 10 monclient(hunting): picked mon.noname-a con 0x7fe6540ad1e0 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:52.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe65c5ab640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6541d7130 con 0x7fe6540ad1e0 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe65c5ab640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe654139b10 con 0x7fe6541d7cd0 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe65c5ab640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65413ffa0 con 0x7fe65409ac40 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe65c5ab640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe65c5ab640 10 monclient(hunting): authenticate will time out at 887.627038s 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe659b46640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fe6541d7cd0 0x7fe654138cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54270/0 (socket says 172.21.15.177:54270) 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe659b46640 1 -- 172.21.15.177:0/363973368 learned_addr learned my addr 172.21.15.177:0/363973368 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 client.?.objecter ms_handle_connect 0x7fe6541d7cd0 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 client.?.objecter ms_handle_connect 0x7fe6540ad1e0 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372970301 0 0) 0x7fe654139b10 con 0x7fe6541d7cd0 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe638004eb0 con 0x7fe6541d7cd0 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531816642 0 0) 0x7fe6541d7130 con 0x7fe6540ad1e0 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe654139b10 con 0x7fe6540ad1e0 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 client.?.objecter ms_handle_connect 0x7fe65409ac40 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.388+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3343792826 0 0) 0x7fe638004eb0 con 0x7fe6541d7cd0 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6541d7130 con 0x7fe6541d7cd0 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2239035528 0 0) 0x7fe654139b10 con 0x7fe6540ad1e0 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe638004eb0 con 0x7fe6540ad1e0 2024-03-29T10:47:52.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe6440019a0 con 0x7fe6541d7cd0 2024-03-29T10:47:52.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:52.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): dump: 2024-03-29T10:47:52.390 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.390 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.390 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.391 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.391 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.391 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.391 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.391 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.391 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.391 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.392 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:52.392 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe64c001f00 con 0x7fe6540ad1e0 2024-03-29T10:47:52.392 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.392 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:52.392 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): dump: 2024-03-29T10:47:52.392 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323726217 0 0) 0x7fe65413ffa0 con 0x7fe65409ac40 2024-03-29T10:47:52.393 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe654139b10 con 0x7fe65409ac40 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3267066743 0 0) 0x7fe6541d7130 con 0x7fe6541d7cd0 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 >> v1:172.21.15.5:6790/0 conn(0x7fe65409ac40 legacy=0x7fe6541d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 >> v1:172.21.15.5:6789/0 conn(0x7fe6540ad1e0 legacy=0x7fe654139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 monclient: found mon.b 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540ae520 con 0x7fe6541d7cd0 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient: _finish_auth 0 2024-03-29T10:47:52.394 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe65c5ab640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe65c5ab640 10 monclient: _renew_subs 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe65c5ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe65c5ab640 1 -- 172.21.15.177:0/363973368 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6540ae800 con 0x7fe6541d7cd0 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe65c5ab640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe65c5ab640 10 monclient: _renew_subs 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe65c5ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe65c5ab640 1 -- 172.21.15.177:0/363973368 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe654142b70 con 0x7fe6541d7cd0 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe65c5ab640 1 librados: init done 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe644001b00 con 0x7fe6541d7cd0 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe6440038c0 con 0x7fe6541d7cd0 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:52.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 monclient: dump: 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe658b44640 4 set_mon_vals no callback set 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe644003e50 con 0x7fe6541d7cd0 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1471986050 0 0) 0x7fe644006600 con 0x7fe6541d7cd0 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 10 client.4120.objecter ms_dispatch 0x7fe654000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:47:52.396 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:47:52.397 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.389+0000 7fe64affd640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T10:47:52.397 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.390+0000 7fe65c5ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.397 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.390+0000 7fe65c5ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.397 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.390+0000 7fe65c5ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.397 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.390+0000 7fe65c5ab640 1 -- 172.21.15.177:0/363973368 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65409b020 con 0x7fe6541d7cd0 2024-03-29T10:47:52.397 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.392+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe6440043f0 con 0x7fe6541d7cd0 2024-03-29T10:47:52.397 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.392+0000 7fe64affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.397 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.392+0000 7fe64affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:52.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.464+0000 7fe65c5ab640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "79b602b0-e74b-4fa1-8418-09e0e3a40335", "id": 1}] 2024-03-29T10:47:52.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.464+0000 7fe65c5ab640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "79b602b0-e74b-4fa1-8418-09e0e3a40335", "id": 1}] 2024-03-29T10:47:52.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.464+0000 7fe65c5ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.448 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.464+0000 7fe65c5ab640 1 -- 172.21.15.177:0/363973368 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd new", "uuid": "79b602b0-e74b-4fa1-8418-09e0e3a40335", "id": 1} v 0) v1 -- 0x7fe65413df40 con 0x7fe6541d7cd0 2024-03-29T10:47:52.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.469+0000 7fe64affd640 1 -- 172.21.15.177:0/363973368 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "79b602b0-e74b-4fa1-8418-09e0e3a40335", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3359897994 0 940483276) 0x7fe644005890 con 0x7fe6541d7cd0 2024-03-29T10:47:52.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.469+0000 7fe64affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "79b602b0-e74b-4fa1-8418-09e0e3a40335", "id": 1}] 2024-03-29T10:47:52.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.469+0000 7fe64affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:52.453 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 10 librados: watch_flush enter 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 10 librados: watch_flush exit 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 10 client.4120.objecter successfully canceled tick 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 10 monclient: shutdown 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 1 -- 172.21.15.177:0/363973368 >> v1:172.21.15.177:6789/0 conn(0x7fe6541d7cd0 legacy=0x7fe654138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 1 -- 172.21.15.177:0/363973368 shutdown_connections 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 1 -- 172.21.15.177:0/363973368 >> 172.21.15.177:0/363973368 conn(0x7fe654066b00 msgr2=0x7fe6540b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 1 -- 172.21.15.177:0/363973368 shutdown_connections 2024-03-29T10:47:52.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.470+0000 7fe65c5ab640 1 -- 172.21.15.177:0/363973368 wait complete. 2024-03-29T10:47:52.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.471+0000 7fe65c5ab640 1 librados: shutdown 2024-03-29T10:47:52.463 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 5e501eca-51c0-4012-a928-1985451ebaca 2 2024-03-29T10:47:52.564 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.581+0000 7fb4a3238640 10 monclient: get_monmap_and_config 2024-03-29T10:47:52.564 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.581+0000 7fb4a3238640 10 monclient: build_initial_monmap 2024-03-29T10:47:52.564 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.581+0000 7fb4a3238640 10 monclient: monmap: 2024-03-29T10:47:52.564 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:52.564 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.564 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:52.582889+0000 2024-03-29T10:47:52.565 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:52.582889+0000 2024-03-29T10:47:52.565 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:52.565 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.565 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:52.565 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:52.565 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:52.565 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 1 Processor -- start 2024-03-29T10:47:52.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 1 -- start start 2024-03-29T10:47:52.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 10 monclient: init 2024-03-29T10:47:52.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 10 monclient(hunting): picked mon.noname-c con 0x7fb49c0b1780 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 10 monclient(hunting): picked mon.noname-b con 0x7fb49c13d7e0 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 10 monclient(hunting): picked mon.noname-a con 0x7fb49c143be0 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c070b30 con 0x7fb49c143be0 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c070da0 con 0x7fb49c0b1780 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c09adb0 con 0x7fb49c13d7e0 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a3238640 10 monclient(hunting): authenticate will time out at 887.821484s 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a0fd4640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb49c0b1780 0x7fb49c13d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54280/0 (socket says 172.21.15.177:54280) 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.582+0000 7fb4a0fd4640 1 -- 172.21.15.177:0/2835145580 learned_addr learned my addr 172.21.15.177:0/2835145580 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302954601 0 0) 0x7fb49c070da0 con 0x7fb49c0b1780 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c003520 con 0x7fb49c0b1780 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301553439 0 0) 0x7fb49c070b30 con 0x7fb49c143be0 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c070da0 con 0x7fb49c143be0 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757734128 0 0) 0x7fb49c09adb0 con 0x7fb49c13d7e0 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:47:52.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c070b30 con 0x7fb49c13d7e0 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 509952473 0 0) 0x7fb48c003520 con 0x7fb49c0b1780 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 >> v1:172.21.15.5:6790/0 conn(0x7fb49c13d7e0 legacy=0x7fb49c1434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 >> v1:172.21.15.5:6789/0 conn(0x7fb49c143be0 legacy=0x7fb49c14b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 monclient: found mon.noname-c 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c144070 con 0x7fb49c0b1780 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb490001690 con 0x7fb49c0b1780 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb4a3238640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:47:52.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: dump: 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb4a3238640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb490001870 con 0x7fb49c0b1780 2024-03-29T10:47:52.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 1 -- 172.21.15.177:0/2835145580 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb490001c80 con 0x7fb49c0b1780 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb4a3238640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb4a3238640 4 set_mon_vals no callback set 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.583+0000 7fb49b7fe640 10 monclient: dump: 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.579 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.584+0000 7fb4a3238640 10 monclient: shutdown 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.584+0000 7fb4a3238640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.584+0000 7fb4a3238640 1 -- 172.21.15.177:0/2835145580 >> v1:172.21.15.177:6789/0 conn(0x7fb49c0b1780 legacy=0x7fb49c13d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.584+0000 7fb4a3238640 1 -- 172.21.15.177:0/2835145580 shutdown_connections 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.584+0000 7fb4a3238640 1 -- 172.21.15.177:0/2835145580 >> 172.21.15.177:0/2835145580 conn(0x7fb49c0b58f0 msgr2=0x7fb49c0b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.584+0000 7fb4a3238640 1 -- 172.21.15.177:0/2835145580 shutdown_connections 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.584+0000 7fb4a3238640 1 -- 172.21.15.177:0/2835145580 wait complete. 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.584+0000 7fb4a3238640 10 monclient: build_initial_monmap 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.584+0000 7fb4a3238640 10 monclient: monmap: 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:52.580 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 1 librados: starting msgr at 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 1 librados: starting objecter 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 1 Processor -- start 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 1 -- start start 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 1 librados: setting wanted keys 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 1 librados: calling monclient init 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 10 monclient: init 2024-03-29T10:47:52.581 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 10 monclient(hunting): picked mon.noname-b con 0x7fb49c0b1780 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 10 monclient(hunting): picked mon.noname-a con 0x7fb49c13d7e0 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 10 monclient(hunting): picked mon.noname-c con 0x7fb49c143be0 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c137ed0 con 0x7fb49c13d7e0 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c1d5760 con 0x7fb49c0b1780 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.585+0000 7fb4a3238640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c1d53f0 con 0x7fb49c143be0 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 10 monclient(hunting): authenticate will time out at 887.824630s 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a0fd4640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb49c0b1780 0x7fb49c1d4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54294/0 (socket says 172.21.15.177:54294) 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a0fd4640 1 -- 172.21.15.177:0/700967967 learned_addr learned my addr 172.21.15.177:0/700967967 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 client.?.objecter ms_handle_connect 0x7fb49c0b1780 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.582 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 client.?.objecter ms_handle_connect 0x7fb49c13d7e0 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 client.?.objecter ms_handle_connect 0x7fb49c143be0 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121023586 0 0) 0x7fb49c1d5760 con 0x7fb49c0b1780 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb480004eb0 con 0x7fb49c0b1780 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879771617 0 0) 0x7fb49c137ed0 con 0x7fb49c13d7e0 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:47:52.583 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c1d5760 con 0x7fb49c13d7e0 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139095572 0 0) 0x7fb49c1d53f0 con 0x7fb49c143be0 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c137ed0 con 0x7fb49c143be0 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4138863921 0 0) 0x7fb480004eb0 con 0x7fb49c0b1780 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb49c1d53f0 con 0x7fb49c0b1780 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb490001bc0 con 0x7fb49c0b1780 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): dump: 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.584 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3113437608 0 0) 0x7fb49c1d5760 con 0x7fb49c13d7e0 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb480004eb0 con 0x7fb49c13d7e0 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb488001870 con 0x7fb49c13d7e0 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:52.585 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): dump: 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2590281241 0 0) 0x7fb49c137ed0 con 0x7fb49c143be0 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb49c1d5760 con 0x7fb49c143be0 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb494001a70 con 0x7fb49c143be0 2024-03-29T10:47:52.586 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient(hunting): dump: 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3347893513 0 0) 0x7fb49c1d53f0 con 0x7fb49c0b1780 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 >> v1:172.21.15.5:6790/0 conn(0x7fb49c143be0 legacy=0x7fb49c1d81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.587 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 >> v1:172.21.15.5:6789/0 conn(0x7fb49c13d7e0 legacy=0x7fb49c132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 monclient: found mon.b 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c142500 con 0x7fb49c0b1780 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 10 monclient: _renew_subs 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c142750 con 0x7fb49c0b1780 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:52.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 10 monclient: _renew_subs 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49c133260 con 0x7fb49c0b1780 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4a3238640 1 librados: init done 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb490004bf0 con 0x7fb49c0b1780 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb490004f00 con 0x7fb49c0b1780 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb49affd640 4 set_mon_vals no callback set 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.586+0000 7fb4997fa640 10 monclient: dump: 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.589 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.587+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb4900053f0 con 0x7fb49c0b1780 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.587+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3344426615 0 0) 0x7fb490005ac0 con 0x7fb49c0b1780 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.587+0000 7fb4997fa640 10 client.4126.objecter ms_dispatch 0x7fb49c000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.587+0000 7fb4997fa640 3 client.4126.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.587+0000 7fb4997fa640 3 client.4126.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.587+0000 7fb4997fa640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.587+0000 7fb4a3238640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.587+0000 7fb4a3238640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.587+0000 7fb4a3238640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.587+0000 7fb4a3238640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c005080 con 0x7fb49c0b1780 2024-03-29T10:47:52.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.589+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb4900056a0 con 0x7fb49c0b1780 2024-03-29T10:47:52.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.589+0000 7fb4997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.589+0000 7fb4997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:52.636 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:52.653+0000 7f4f87a17040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:47:52.644 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.662+0000 7fb4a3238640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5e501eca-51c0-4012-a928-1985451ebaca", "id": 2}] 2024-03-29T10:47:52.645 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.662+0000 7fb4a3238640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5e501eca-51c0-4012-a928-1985451ebaca", "id": 2}] 2024-03-29T10:47:52.645 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.662+0000 7fb4a3238640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.645 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.662+0000 7fb4a3238640 1 -- 172.21.15.177:0/700967967 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5e501eca-51c0-4012-a928-1985451ebaca", "id": 2} v 0) v1 -- 0x7fb47c0058b0 con 0x7fb49c0b1780 2024-03-29T10:47:52.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.669+0000 7fb4997fa640 1 -- 172.21.15.177:0/700967967 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5e501eca-51c0-4012-a928-1985451ebaca", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1823986064 0 216599125) 0x7fb490005c80 con 0x7fb49c0b1780 2024-03-29T10:47:52.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.669+0000 7fb4997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5e501eca-51c0-4012-a928-1985451ebaca", "id": 2}] 2024-03-29T10:47:52.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.669+0000 7fb4997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:52.652 INFO:teuthology.orchestra.run.smithi177.stdout:2 2024-03-29T10:47:52.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 10 librados: watch_flush enter 2024-03-29T10:47:52.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 10 librados: watch_flush exit 2024-03-29T10:47:52.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T10:47:52.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 10 client.4126.objecter successfully canceled tick 2024-03-29T10:47:52.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 10 monclient: shutdown 2024-03-29T10:47:52.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 1 -- 172.21.15.177:0/700967967 >> v1:172.21.15.177:6789/0 conn(0x7fb49c0b1780 legacy=0x7fb49c1d4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 1 -- 172.21.15.177:0/700967967 shutdown_connections 2024-03-29T10:47:52.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 1 -- 172.21.15.177:0/700967967 >> 172.21.15.177:0/700967967 conn(0x7fb49c0b58f0 msgr2=0x7fb49c069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 1 -- 172.21.15.177:0/700967967 shutdown_connections 2024-03-29T10:47:52.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 1 -- 172.21.15.177:0/700967967 wait complete. 2024-03-29T10:47:52.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.670+0000 7fb4a3238640 1 librados: shutdown 2024-03-29T10:47:52.658 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:52.676+0000 7fc1aea42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:47:52.663 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 867c7e71-84e2-4983-8560-f9f830e2c1ec 3 2024-03-29T10:47:52.694 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:52.711+0000 7f4f87a17040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:47:52.714 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:52.732+0000 7fc1aea42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:47:52.741 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:52.758+0000 7f4f87a17040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:47:52.761 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:52.778+0000 7fc1aea42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:47:52.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.781+0000 7fb261eae640 10 monclient: get_monmap_and_config 2024-03-29T10:47:52.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.781+0000 7fb261eae640 10 monclient: build_initial_monmap 2024-03-29T10:47:52.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.781+0000 7fb261eae640 10 monclient: monmap: 2024-03-29T10:47:52.763 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:52.763 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.763 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:52.782496+0000 2024-03-29T10:47:52.763 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:52.782496+0000 2024-03-29T10:47:52.764 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:52.764 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.764 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:52.764 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:52.764 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:52.764 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.781+0000 7fb261eae640 1 Processor -- start 2024-03-29T10:47:52.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 1 -- start start 2024-03-29T10:47:52.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 10 monclient: init 2024-03-29T10:47:52.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:52.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:52.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 10 monclient(hunting): picked mon.noname-c con 0x7fb25c09ac40 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:52.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 10 monclient(hunting): picked mon.noname-a con 0x7fb25c09c870 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:52.773 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 10 monclient(hunting): picked mon.noname-b con 0x7fb25c0b4360 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c09dec0 con 0x7fb25c09c870 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c09da30 con 0x7fb25c09ac40 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c0b39c0 con 0x7fb25c0b4360 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb261eae640 10 monclient(hunting): authenticate will time out at 888.021333s 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb25b7fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb25c09ac40 0x7fb25c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54308/0 (socket says 172.21.15.177:54308) 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.782+0000 7fb25b7fe640 1 -- 172.21.15.177:0/3347513176 learned_addr learned my addr 172.21.15.177:0/3347513176 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456891430 0 0) 0x7fb25c09da30 con 0x7fb25c09ac40 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb248003520 con 0x7fb25c09ac40 2024-03-29T10:47:52.774 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938207874 0 0) 0x7fb25c0b39c0 con 0x7fb25c0b4360 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb25c09da30 con 0x7fb25c0b4360 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113116247 0 0) 0x7fb244001110 con 0x7fb25c09c870 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb25c0b39c0 con 0x7fb25c09c870 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1455549042 0 0) 0x7fb248003520 con 0x7fb25c09ac40 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 >> v1:172.21.15.5:6790/0 conn(0x7fb25c0b4360 legacy=0x7fb25c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 >> v1:172.21.15.5:6789/0 conn(0x7fb25c09c870 legacy=0x7fb25c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 monclient: found mon.noname-c 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c0b5160 con 0x7fb25c09ac40 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: _finish_auth 0 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb24c001690 con 0x7fb25c09ac40 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb261eae640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb261eae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: dump: 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.776 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb261eae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb24c001870 con 0x7fb25c09ac40 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 1 -- 172.21.15.177:0/3347513176 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb24c001c80 con 0x7fb25c09ac40 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb261eae640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb261eae640 4 set_mon_vals no callback set 2024-03-29T10:47:52.777 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.783+0000 7fb25affd640 10 monclient: dump: 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.784+0000 7fb261eae640 10 monclient: shutdown 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.784+0000 7fb261eae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.778 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.784+0000 7fb261eae640 1 -- 172.21.15.177:0/3347513176 >> v1:172.21.15.177:6789/0 conn(0x7fb25c09ac40 legacy=0x7fb25c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.784+0000 7fb261eae640 1 -- 172.21.15.177:0/3347513176 shutdown_connections 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.784+0000 7fb261eae640 1 -- 172.21.15.177:0/3347513176 >> 172.21.15.177:0/3347513176 conn(0x7fb25c066b00 msgr2=0x7fb25c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.784+0000 7fb261eae640 1 -- 172.21.15.177:0/3347513176 shutdown_connections 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.784+0000 7fb261eae640 1 -- 172.21.15.177:0/3347513176 wait complete. 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.784+0000 7fb261eae640 10 monclient: build_initial_monmap 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.784+0000 7fb261eae640 10 monclient: monmap: 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:52.779 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 1 librados: starting msgr at 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 1 librados: starting objecter 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 1 Processor -- start 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 1 -- start start 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 1 librados: setting wanted keys 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 1 librados: calling monclient init 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 10 monclient: init 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 10 monclient(hunting): picked mon.noname-b con 0x7fb25c09ac40 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 10 monclient(hunting): picked mon.noname-c con 0x7fb25c138f60 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 10 monclient(hunting): picked mon.noname-a con 0x7fb25c0afdc0 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c1db610 con 0x7fb25c0afdc0 2024-03-29T10:47:52.780 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c139a00 con 0x7fb25c09ac40 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c0b0230 con 0x7fb25c138f60 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb261eae640 10 monclient(hunting): authenticate will time out at 888.024372s 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb25b7fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb25c09ac40 0x7fb25c1dac80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54318/0 (socket says 172.21.15.177:54318) 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb25b7fe640 1 -- 172.21.15.177:0/842647527 learned_addr learned my addr 172.21.15.177:0/842647527 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb258ff9640 10 client.?.objecter ms_handle_connect 0x7fb25c09ac40 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb258ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.785+0000 7fb258ff9640 10 client.?.objecter ms_handle_connect 0x7fb25c0afdc0 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258099926 0 0) 0x7fb25c139a00 con 0x7fb25c09ac40 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:47:52.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb23c004eb0 con 0x7fb25c09ac40 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 client.?.objecter ms_handle_connect 0x7fb25c138f60 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356020141 0 0) 0x7fb25c1db610 con 0x7fb25c0afdc0 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb25c139a00 con 0x7fb25c0afdc0 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983975471 0 0) 0x7fb25c0b0230 con 0x7fb25c138f60 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:47:52.782 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb25c1db610 con 0x7fb25c138f60 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189502530 0 0) 0x7fb23c004eb0 con 0x7fb25c09ac40 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb25c0b0230 con 0x7fb25c09ac40 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb24c001bc0 con 0x7fb25c09ac40 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.783 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2313390599 0 0) 0x7fb25c139a00 con 0x7fb25c0afdc0 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb23c004eb0 con 0x7fb25c0afdc0 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb254001a90 con 0x7fb25c0afdc0 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.784 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1156826970 0 0) 0x7fb25c0b0230 con 0x7fb25c09ac40 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 >> v1:172.21.15.5:6790/0 conn(0x7fb25c138f60 legacy=0x7fb25c0af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 >> v1:172.21.15.5:6789/0 conn(0x7fb25c0afdc0 legacy=0x7fb25c1e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 monclient: found mon.b 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c0b03c0 con 0x7fb25c09ac40 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient: _finish_auth 0 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb261eae640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:47:52.785 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb261eae640 10 monclient: _renew_subs 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb261eae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb261eae640 1 -- 172.21.15.177:0/842647527 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25c1e36e0 con 0x7fb25c09ac40 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb261eae640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb261eae640 10 monclient: _renew_subs 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb261eae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb261eae640 1 -- 172.21.15.177:0/842647527 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25c1e3a50 con 0x7fb25c09ac40 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb261eae640 1 librados: init done 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb24c004bf0 con 0x7fb25c09ac40 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb24c004d60 con 0x7fb25c09ac40 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb25a7fc640 4 set_mon_vals no callback set 2024-03-29T10:47:52.786 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 monclient: dump: 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb24c005250 con 0x7fb25c09ac40 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2911564704 0 0) 0x7fb24c005990 con 0x7fb25c09ac40 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 10 client.4132.objecter ms_dispatch 0x7fb25c000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:47:52.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.786+0000 7fb258ff9640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T10:47:52.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.787+0000 7fb261eae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.787+0000 7fb261eae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.787+0000 7fb261eae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.787+0000 7fb261eae640 1 -- 172.21.15.177:0/842647527 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb240005080 con 0x7fb25c09ac40 2024-03-29T10:47:52.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.789+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb24c005b50 con 0x7fb25c09ac40 2024-03-29T10:47:52.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.789+0000 7fb258ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.789+0000 7fb258ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:52.798 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:52.815+0000 7f4f87a17040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:47:52.816 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:52.834+0000 7fc1aea42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:47:52.842 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:52.859+0000 7f4f87a17040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:47:52.844 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.862+0000 7fb261eae640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "867c7e71-84e2-4983-8560-f9f830e2c1ec", "id": 3}] 2024-03-29T10:47:52.845 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.862+0000 7fb261eae640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "867c7e71-84e2-4983-8560-f9f830e2c1ec", "id": 3}] 2024-03-29T10:47:52.845 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.862+0000 7fb261eae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:52.845 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.862+0000 7fb261eae640 1 -- 172.21.15.177:0/842647527 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd new", "uuid": "867c7e71-84e2-4983-8560-f9f830e2c1ec", "id": 3} v 0) v1 -- 0x7fb2400058b0 con 0x7fb25c09ac40 2024-03-29T10:47:52.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.869+0000 7fb258ff9640 1 -- 172.21.15.177:0/842647527 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "867c7e71-84e2-4983-8560-f9f830e2c1ec", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2466841067 0 525046306) 0x7fb24c005d70 con 0x7fb25c09ac40 2024-03-29T10:47:52.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.869+0000 7fb258ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "867c7e71-84e2-4983-8560-f9f830e2c1ec", "id": 3}] 2024-03-29T10:47:52.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.869+0000 7fb258ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:52.852 INFO:teuthology.orchestra.run.smithi177.stdout:3 2024-03-29T10:47:52.853 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 10 librados: watch_flush enter 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 10 librados: watch_flush exit 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 10 client.4132.objecter successfully canceled tick 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 10 monclient: shutdown 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 1 -- 172.21.15.177:0/842647527 >> v1:172.21.15.177:6789/0 conn(0x7fb25c09ac40 legacy=0x7fb25c1dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 1 -- 172.21.15.177:0/842647527 shutdown_connections 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 1 -- 172.21.15.177:0/842647527 >> 172.21.15.177:0/842647527 conn(0x7fb25c066b00 msgr2=0x7fb25c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 1 -- 172.21.15.177:0/842647527 shutdown_connections 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 1 -- 172.21.15.177:0/842647527 wait complete. 2024-03-29T10:47:52.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:52.870+0000 7fb261eae640 1 librados: shutdown 2024-03-29T10:47:52.861 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:52.879+0000 7fc1aea42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:47:52.863 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0 4 2024-03-29T10:47:52.963 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:52.981+0000 7f4f87a17040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:47:52.981 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:52.999+0000 7fc1aea42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:47:53.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.020+0000 7fb8b38e4640 10 monclient: get_monmap_and_config 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.020+0000 7fb8b38e4640 10 monclient: build_initial_monmap 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.020+0000 7fb8b38e4640 10 monclient: monmap: 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:53.021746+0000 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:53.021746+0000 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:53.003 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.014 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.020+0000 7fb8b38e4640 1 Processor -- start 2024-03-29T10:47:53.014 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 1 -- start start 2024-03-29T10:47:53.014 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 10 monclient: init 2024-03-29T10:47:53.014 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:53.014 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 10 monclient(hunting): picked mon.noname-a con 0x7fb8ac09ac40 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 10 monclient(hunting): picked mon.noname-c con 0x7fb8ac09c870 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 10 monclient(hunting): picked mon.noname-b con 0x7fb8ac0b4360 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac09dec0 con 0x7fb8ac09ac40 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac09da30 con 0x7fb8ac09c870 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac0b39c0 con 0x7fb8ac0b4360 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b38e4640 10 monclient(hunting): authenticate will time out at 888.260344s 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b0e7f640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb8ac09c870 0x7fb8ac13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54332/0 (socket says 172.21.15.177:54332) 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.021+0000 7fb8b0e7f640 1 -- 172.21.15.177:0/3633599916 learned_addr learned my addr 172.21.15.177:0/3633599916 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:53.015 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681940552 0 0) 0x7fb8ac09da30 con 0x7fb8ac09c870 2024-03-29T10:47:53.016 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.016 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.016 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.016 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:47:53.016 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb898003520 con 0x7fb8ac09c870 2024-03-29T10:47:53.016 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258034241 0 0) 0x7fb8ac0b39c0 con 0x7fb8ac0b4360 2024-03-29T10:47:53.016 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.016 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.016 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.016 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ac09da30 con 0x7fb8ac0b4360 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903043080 0 0) 0x7fb8a8001110 con 0x7fb8ac09ac40 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ac0b39c0 con 0x7fb8ac09ac40 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 117906923 0 0) 0x7fb898003520 con 0x7fb8ac09c870 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 >> v1:172.21.15.5:6790/0 conn(0x7fb8ac0b4360 legacy=0x7fb8ac14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 >> v1:172.21.15.5:6789/0 conn(0x7fb8ac09ac40 legacy=0x7fb8ac0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:53.017 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 monclient: found mon.noname-c 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0b5160 con 0x7fb8ac09c870 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb89c001690 con 0x7fb8ac09c870 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: dump: 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.018 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.019 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.019 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.019 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.019 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.019 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.019 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.019 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8b38e4640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:47:53.019 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8b38e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:53.019 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb89c001870 con 0x7fb8ac09c870 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 1 -- 172.21.15.177:0/3633599916 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb89c001c80 con 0x7fb8ac09c870 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8b38e4640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8b38e4640 4 set_mon_vals no callback set 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8a3fff640 10 monclient: dump: 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.020 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8b38e4640 10 monclient: shutdown 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8b38e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.022+0000 7fb8b38e4640 1 -- 172.21.15.177:0/3633599916 >> v1:172.21.15.177:6789/0 conn(0x7fb8ac09c870 legacy=0x7fb8ac13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.023+0000 7fb8b38e4640 1 -- 172.21.15.177:0/3633599916 shutdown_connections 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.023+0000 7fb8b38e4640 1 -- 172.21.15.177:0/3633599916 >> 172.21.15.177:0/3633599916 conn(0x7fb8ac066b00 msgr2=0x7fb8ac066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.023+0000 7fb8b38e4640 1 -- 172.21.15.177:0/3633599916 shutdown_connections 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.023+0000 7fb8b38e4640 1 -- 172.21.15.177:0/3633599916 wait complete. 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.023+0000 7fb8b38e4640 10 monclient: build_initial_monmap 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.023+0000 7fb8b38e4640 10 monclient: monmap: 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:53.021 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.023+0000 7fb8b38e4640 1 librados: starting msgr at 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.023+0000 7fb8b38e4640 1 librados: starting objecter 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 1 Processor -- start 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 1 -- start start 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 1 librados: setting wanted keys 2024-03-29T10:47:53.022 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 1 librados: calling monclient init 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 10 monclient: init 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 10 monclient(hunting): picked mon.noname-c con 0x7fb8ac09ac40 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 10 monclient(hunting): picked mon.noname-a con 0x7fb8ac0acd00 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 10 monclient(hunting): picked mon.noname-b con 0x7fb8ac139110 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac1d7520 con 0x7fb8ac0acd00 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac139c30 con 0x7fb8ac139110 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac0b1ad0 con 0x7fb8ac09ac40 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b38e4640 10 monclient(hunting): authenticate will time out at 888.263227s 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b1e81640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb8ac139110 0x7fb8ac139520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54338/0 (socket says 172.21.15.177:54338) 2024-03-29T10:47:53.023 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8b1e81640 1 -- 172.21.15.177:0/4273942799 learned_addr learned my addr 172.21.15.177:0/4273942799 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 client.?.objecter ms_handle_connect 0x7fb8ac139110 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 client.?.objecter ms_handle_connect 0x7fb8ac0acd00 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 client.?.objecter ms_handle_connect 0x7fb8ac09ac40 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026887120 0 0) 0x7fb8ac139c30 con 0x7fb8ac139110 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb890004eb0 con 0x7fb8ac139110 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311423211 0 0) 0x7fb8ac1d7520 con 0x7fb8ac0acd00 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.024 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ac139c30 con 0x7fb8ac0acd00 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922131176 0 0) 0x7fb8ac0b1ad0 con 0x7fb8ac09ac40 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.024+0000 7fb8a1ffb640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ac1d7520 con 0x7fb8ac09ac40 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1103659968 0 0) 0x7fb890004eb0 con 0x7fb8ac139110 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8ac0b1ad0 con 0x7fb8ac139110 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb8a4001c00 con 0x7fb8ac139110 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:53.025 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.026 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1555203460 0 0) 0x7fb8ac139c30 con 0x7fb8ac0acd00 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb890004eb0 con 0x7fb8ac0acd00 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb89c001b80 con 0x7fb8ac0acd00 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.027 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2725897540 0 0) 0x7fb8ac1d7520 con 0x7fb8ac09ac40 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8ac139c30 con 0x7fb8ac09ac40 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb8a8001860 con 0x7fb8ac09ac40 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.028 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4032034181 0 0) 0x7fb8ac0b1ad0 con 0x7fb8ac139110 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 >> v1:172.21.15.5:6790/0 conn(0x7fb8ac09ac40 legacy=0x7fb8ac0b0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 >> v1:172.21.15.5:6789/0 conn(0x7fb8ac0acd00 legacy=0x7fb8ac1d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 monclient: found mon.b 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0b0860 con 0x7fb8ac139110 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:53.029 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8b38e4640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8b38e4640 10 monclient: _renew_subs 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8b38e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8b38e4640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac0af6e0 con 0x7fb8ac139110 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8a40020d0 con 0x7fb8ac139110 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fb8a4003b30 con 0x7fb8ac139110 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 monclient: dump: 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.030 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8b38e4640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8b38e4640 10 monclient: _renew_subs 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a37fe640 4 set_mon_vals no callback set 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8b38e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8b38e4640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac1d80c0 con 0x7fb8ac139110 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb8a40040a0 con 0x7fb8ac139110 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8b38e4640 1 librados: init done 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 342076800 0 0) 0x7fb8a4004350 con 0x7fb8ac139110 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 10 client.4138.objecter ms_dispatch 0x7fb8ac000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:47:53.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:47:53.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:47:53.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.025+0000 7fb8a1ffb640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T10:47:53.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.026+0000 7fb8b38e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.026+0000 7fb8b38e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.026+0000 7fb8b38e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.026+0000 7fb8b38e4640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac09b020 con 0x7fb8ac139110 2024-03-29T10:47:53.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.028+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb8a4004510 con 0x7fb8ac139110 2024-03-29T10:47:53.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.028+0000 7fb8a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.028+0000 7fb8a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:53.037 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:53.054+0000 7f4f87a17040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:47:53.054 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:53.072+0000 7fc1aea42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:47:53.082 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.099+0000 7fb8b38e4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0", "id": 4}] 2024-03-29T10:47:53.082 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.099+0000 7fb8b38e4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0", "id": 4}] 2024-03-29T10:47:53.082 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.099+0000 7fb8b38e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.082 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.099+0000 7fb8b38e4640 1 -- 172.21.15.177:0/4273942799 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0", "id": 4} v 0) v1 -- 0x7fb8ac04f730 con 0x7fb8ac139110 2024-03-29T10:47:53.082 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:53.100+0000 7f4f87a17040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:47:53.089 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.106+0000 7fb8a1ffb640 1 -- 172.21.15.177:0/4273942799 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 845114508 0 1697012583) 0x7fb8a4005b40 con 0x7fb8ac139110 2024-03-29T10:47:53.089 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.106+0000 7fb8a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0", "id": 4}] 2024-03-29T10:47:53.089 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.106+0000 7fb8a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:53.090 INFO:teuthology.orchestra.run.smithi177.stdout:4 2024-03-29T10:47:53.091 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.107+0000 7fb8b38e4640 10 librados: watch_flush enter 2024-03-29T10:47:53.091 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.107+0000 7fb8b38e4640 10 librados: watch_flush exit 2024-03-29T10:47:53.091 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.107+0000 7fb8b38e4640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T10:47:53.091 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.107+0000 7fb8b38e4640 10 client.4138.objecter successfully canceled tick 2024-03-29T10:47:53.091 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.107+0000 7fb8b38e4640 10 monclient: shutdown 2024-03-29T10:47:53.093 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.107+0000 7fb8b38e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:53.093 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.107+0000 7fb8b38e4640 1 -- 172.21.15.177:0/4273942799 >> v1:172.21.15.177:6789/0 conn(0x7fb8ac139110 legacy=0x7fb8ac139520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.093 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.107+0000 7fb8b38e4640 1 -- 172.21.15.177:0/4273942799 shutdown_connections 2024-03-29T10:47:53.093 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.107+0000 7fb8b38e4640 1 -- 172.21.15.177:0/4273942799 >> 172.21.15.177:0/4273942799 conn(0x7fb8ac066b00 msgr2=0x7fb8ac13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:53.093 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.108+0000 7fb8b38e4640 1 -- 172.21.15.177:0/4273942799 shutdown_connections 2024-03-29T10:47:53.093 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.108+0000 7fb8b38e4640 1 -- 172.21.15.177:0/4273942799 wait complete. 2024-03-29T10:47:53.093 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.108+0000 7fb8b38e4640 1 librados: shutdown 2024-03-29T10:47:53.101 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new d8b7c8ae-e849-48e3-b698-eddd18808d4c 5 2024-03-29T10:47:53.102 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:53.120+0000 7fc1aea42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:47:53.139 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:53.157+0000 7f4f87a17040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:47:53.156 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:53.174+0000 7fc1aea42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:47:53.188 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:53.206+0000 7f4f87a17040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.218+0000 7ff0ddd1c640 10 monclient: get_monmap_and_config 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.218+0000 7ff0ddd1c640 10 monclient: build_initial_monmap 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.218+0000 7ff0ddd1c640 10 monclient: monmap: 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:53.219977+0000 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:53.219977+0000 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:53.201 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:53.202 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.208 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:53.224+0000 7fc1aea42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:47:53.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 1 Processor -- start 2024-03-29T10:47:53.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 1 -- start start 2024-03-29T10:47:53.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 10 monclient: init 2024-03-29T10:47:53.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:53.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff0d809ac40 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff0d809c870 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff0d80b4360 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d809dec0 con 0x7ff0d80b4360 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d809da30 con 0x7ff0d809ac40 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d80b39c0 con 0x7ff0d809c870 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0ddd1c640 10 monclient(hunting): authenticate will time out at 888.458475s 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0d77fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7ff0d809ac40 0x7ff0d80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54348/0 (socket says 172.21.15.177:54348) 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.219+0000 7ff0d77fe640 1 -- 172.21.15.177:0/2224592569 learned_addr learned my addr 172.21.15.177:0/2224592569 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409633091 0 0) 0x7ff0d809da30 con 0x7ff0d809ac40 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c0003520 con 0x7ff0d809ac40 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686453979 0 0) 0x7ff0d0001490 con 0x7ff0d80b4360 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d809da30 con 0x7ff0d80b4360 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608446640 0 0) 0x7ff0d80b39c0 con 0x7ff0d809c870 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d0001490 con 0x7ff0d809c870 2024-03-29T10:47:53.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 75358538 0 0) 0x7ff0c0003520 con 0x7ff0d809ac40 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 >> v1:172.21.15.5:6790/0 conn(0x7ff0d809c870 legacy=0x7ff0d813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 >> v1:172.21.15.5:6789/0 conn(0x7ff0d80b4360 legacy=0x7ff0d814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 monclient: found mon.noname-c 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d80b5160 con 0x7ff0d809ac40 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff0c8001690 con 0x7ff0d809ac40 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: dump: 2024-03-29T10:47:53.214 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0ddd1c640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0ddd1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0c8001870 con 0x7ff0d809ac40 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0ddd1c640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:53.215 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 1 -- 172.21.15.177:0/2224592569 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff0c8001c80 con 0x7ff0d809ac40 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0ddd1c640 4 set_mon_vals no callback set 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.220+0000 7ff0d67fc640 10 monclient: dump: 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.216 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.221+0000 7ff0ddd1c640 10 monclient: shutdown 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.221+0000 7ff0ddd1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.221+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/2224592569 >> v1:172.21.15.177:6789/0 conn(0x7ff0d809ac40 legacy=0x7ff0d80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.221+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/2224592569 shutdown_connections 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.221+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/2224592569 >> 172.21.15.177:0/2224592569 conn(0x7ff0d8066b00 msgr2=0x7ff0d8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.221+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/2224592569 shutdown_connections 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.221+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/2224592569 wait complete. 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.221+0000 7ff0ddd1c640 10 monclient: build_initial_monmap 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.221+0000 7ff0ddd1c640 10 monclient: monmap: 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:53.217 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 1 librados: starting msgr at 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 1 librados: starting objecter 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 1 Processor -- start 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 1 -- start start 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 1 librados: setting wanted keys 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 1 librados: calling monclient init 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 10 monclient: init 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff0d809ac40 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:53.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff0d80b1860 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff0d813b5b0 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d81d7660 con 0x7ff0d813b5b0 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d80ade20 con 0x7ff0d80b1860 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d81406b0 con 0x7ff0d809ac40 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0ddd1c640 10 monclient(hunting): authenticate will time out at 888.461516s 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.222+0000 7ff0d6ffd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7ff0d80b1860 0x7ff0d813aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54360/0 (socket says 172.21.15.177:54360) 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0d6ffd640 1 -- 172.21.15.177:0/1256504977 learned_addr learned my addr 172.21.15.177:0/1256504977 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 client.?.objecter ms_handle_connect 0x7ff0d80b1860 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 client.?.objecter ms_handle_connect 0x7ff0d809ac40 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 client.?.objecter ms_handle_connect 0x7ff0d813b5b0 2024-03-29T10:47:53.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235713521 0 0) 0x7ff0d80ade20 con 0x7ff0d80b1860 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c4004eb0 con 0x7ff0d80b1860 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042756686 0 0) 0x7ff0d81406b0 con 0x7ff0d809ac40 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d80ade20 con 0x7ff0d809ac40 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209447945 0 0) 0x7ff0d81d7660 con 0x7ff0d813b5b0 2024-03-29T10:47:53.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d81406b0 con 0x7ff0d813b5b0 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1619266094 0 0) 0x7ff0c4004eb0 con 0x7ff0d80b1860 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0d81d7660 con 0x7ff0d80b1860 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff0cc0019b0 con 0x7ff0d80b1860 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): dump: 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.221 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2640015293 0 0) 0x7ff0d80ade20 con 0x7ff0d809ac40 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0c4004eb0 con 0x7ff0d809ac40 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff0c8001b80 con 0x7ff0d809ac40 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): dump: 2024-03-29T10:47:53.222 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3192559739 0 0) 0x7ff0d81406b0 con 0x7ff0d813b5b0 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0d80ade20 con 0x7ff0d813b5b0 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff0d0001a40 con 0x7ff0d813b5b0 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.223 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient(hunting): dump: 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4260463160 0 0) 0x7ff0d81d7660 con 0x7ff0d80b1860 2024-03-29T10:47:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 >> v1:172.21.15.5:6790/0 conn(0x7ff0d809ac40 legacy=0x7ff0d813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 >> v1:172.21.15.5:6789/0 conn(0x7ff0d813b5b0 legacy=0x7ff0d81e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 monclient: found mon.b 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d81cccd0 con 0x7ff0d80b1860 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient: _finish_auth 0 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0ddd1c640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0ddd1c640 10 monclient: _renew_subs 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0ddd1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d81cce60 con 0x7ff0d80b1860 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0ddd1c640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0ddd1c640 10 monclient: _renew_subs 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0ddd1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d81d7190 con 0x7ff0d80b1860 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0ddd1c640 1 librados: init done 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0cc003610 con 0x7ff0d80b1860 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff0cc003920 con 0x7ff0d80b1860 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0d5ffb640 4 set_mon_vals no callback set 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 monclient: dump: 2024-03-29T10:47:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff0cc003ed0 con 0x7ff0d80b1860 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2301426942 0 0) 0x7ff0cc006820 con 0x7ff0d80b1860 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 10 client.4144.objecter ms_dispatch 0x7ff0d8000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:47:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:47:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.223+0000 7ff0bffff640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T10:47:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.224+0000 7ff0ddd1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.224+0000 7ff0ddd1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.224+0000 7ff0ddd1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.224+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b8005080 con 0x7ff0d80b1860 2024-03-29T10:47:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.226+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff0cc0043b0 con 0x7ff0d80b1860 2024-03-29T10:47:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.226+0000 7ff0bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.226+0000 7ff0bffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:53.281 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.299+0000 7ff0ddd1c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d8b7c8ae-e849-48e3-b698-eddd18808d4c", "id": 5}] 2024-03-29T10:47:53.281 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.299+0000 7ff0ddd1c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d8b7c8ae-e849-48e3-b698-eddd18808d4c", "id": 5}] 2024-03-29T10:47:53.282 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.299+0000 7ff0ddd1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.282 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.299+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/1256504977 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d8b7c8ae-e849-48e3-b698-eddd18808d4c", "id": 5} v 0) v1 -- 0x7ff0b80058b0 con 0x7ff0d80b1860 2024-03-29T10:47:53.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.306+0000 7ff0bffff640 1 -- 172.21.15.177:0/1256504977 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d8b7c8ae-e849-48e3-b698-eddd18808d4c", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1774423662 0 1988412176) 0x7ff0cc006a30 con 0x7ff0d80b1860 2024-03-29T10:47:53.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.306+0000 7ff0bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d8b7c8ae-e849-48e3-b698-eddd18808d4c", "id": 5}] 2024-03-29T10:47:53.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.306+0000 7ff0bffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:53.289 INFO:teuthology.orchestra.run.smithi177.stdout:5 2024-03-29T10:47:53.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 10 librados: watch_flush enter 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 10 librados: watch_flush exit 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 10 client.4144.objecter successfully canceled tick 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 10 monclient: shutdown 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/1256504977 >> v1:172.21.15.177:6789/0 conn(0x7ff0d80b1860 legacy=0x7ff0d813aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/1256504977 shutdown_connections 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/1256504977 >> 172.21.15.177:0/1256504977 conn(0x7ff0d8066b00 msgr2=0x7ff0d813cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/1256504977 shutdown_connections 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 1 -- 172.21.15.177:0/1256504977 wait complete. 2024-03-29T10:47:53.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.307+0000 7ff0ddd1c640 1 librados: shutdown 2024-03-29T10:47:53.300 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new f863ddfb-54f0-4042-8f43-5355132f26af 6 2024-03-29T10:47:53.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.458+0000 7fbdefe3e640 10 monclient: get_monmap_and_config 2024-03-29T10:47:53.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.458+0000 7fbdefe3e640 10 monclient: build_initial_monmap 2024-03-29T10:47:53.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.458+0000 7fbdefe3e640 10 monclient: monmap: 2024-03-29T10:47:53.441 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:53.441 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.441 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:53.459881+0000 2024-03-29T10:47:53.441 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:53.459881+0000 2024-03-29T10:47:53.442 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:53.442 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.442 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:53.442 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:53.442 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:53.442 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.458+0000 7fbdefe3e640 1 Processor -- start 2024-03-29T10:47:53.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 1 -- start start 2024-03-29T10:47:53.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 10 monclient: init 2024-03-29T10:47:53.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:53.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:53.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 10 monclient(hunting): picked mon.noname-c con 0x7fbde809ac40 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 10 monclient(hunting): picked mon.noname-a con 0x7fbde809c870 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 10 monclient(hunting): picked mon.noname-b con 0x7fbde80b4360 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde809dec0 con 0x7fbde809c870 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde809da30 con 0x7fbde809ac40 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde80b39c0 con 0x7fbde80b4360 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdefe3e640 10 monclient(hunting): authenticate will time out at 888.698499s 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.459+0000 7fbdedbda640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fbde809ac40 0x7fbde80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54364/0 (socket says 172.21.15.177:54364) 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdedbda640 1 -- 172.21.15.177:0/3799200958 learned_addr learned my addr 172.21.15.177:0/3799200958 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641521875 0 0) 0x7fbde809da30 con 0x7fbde809ac40 2024-03-29T10:47:53.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd4003520 con 0x7fbde809ac40 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565013202 0 0) 0x7fbdd8001110 con 0x7fbde809c870 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde809da30 con 0x7fbde809c870 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818544567 0 0) 0x7fbde80b39c0 con 0x7fbde80b4360 2024-03-29T10:47:53.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:47:53.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8001110 con 0x7fbde80b4360 2024-03-29T10:47:53.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1436065231 0 0) 0x7fbdd4003520 con 0x7fbde809ac40 2024-03-29T10:47:53.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 >> v1:172.21.15.5:6790/0 conn(0x7fbde80b4360 legacy=0x7fbde814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 >> v1:172.21.15.5:6789/0 conn(0x7fbde809c870 legacy=0x7fbde813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: _finish_hunting 0 2024-03-29T10:47:53.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 monclient: found mon.noname-c 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde80b5160 con 0x7fbde809ac40 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: _finish_auth 0 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: _check_auth_tickets 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdefe3e640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdefe3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbde4001690 con 0x7fbde809ac40 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:53.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: dump: 2024-03-29T10:47:53.457 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.457 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.457 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.457 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.457 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.457 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.457 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.457 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.457 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.458 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.458 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdefe3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:53.458 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbde4001870 con 0x7fbde809ac40 2024-03-29T10:47:53.458 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:53.458 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 1 -- 172.21.15.177:0/3799200958 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbde4001c80 con 0x7fbde809ac40 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdefe3e640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdefe3e640 4 set_mon_vals no callback set 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.460+0000 7fbdecbd8640 10 monclient: dump: 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.459 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.460 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.460 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.460 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.461+0000 7fbdefe3e640 10 monclient: shutdown 2024-03-29T10:47:53.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.461+0000 7fbdefe3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:53.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.461+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3799200958 >> v1:172.21.15.177:6789/0 conn(0x7fbde809ac40 legacy=0x7fbde80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.461+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3799200958 shutdown_connections 2024-03-29T10:47:53.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.461+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3799200958 >> 172.21.15.177:0/3799200958 conn(0x7fbde8066b00 msgr2=0x7fbde8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:53.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.461+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3799200958 shutdown_connections 2024-03-29T10:47:53.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.461+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3799200958 wait complete. 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.462+0000 7fbdefe3e640 10 monclient: build_initial_monmap 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.462+0000 7fbdefe3e640 10 monclient: monmap: 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:53.461 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.462+0000 7fbdefe3e640 1 librados: starting msgr at 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.462+0000 7fbdefe3e640 1 librados: starting objecter 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.462+0000 7fbdefe3e640 1 Processor -- start 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.462+0000 7fbdefe3e640 1 -- start start 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.462+0000 7fbdefe3e640 1 librados: setting wanted keys 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.462+0000 7fbdefe3e640 1 librados: calling monclient init 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.462+0000 7fbdefe3e640 10 monclient: init 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdefe3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdefe3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:53.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdefe3e640 10 monclient(hunting): picked mon.noname-a con 0x7fbde809ac40 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdefe3e640 10 monclient(hunting): picked mon.noname-b con 0x7fbde81ccb70 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdefe3e640 10 monclient(hunting): picked mon.noname-c con 0x7fbde81425f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdefe3e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde81cd520 con 0x7fbde809ac40 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdefe3e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde80acd00 con 0x7fbde81ccb70 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdefe3e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde81cd000 con 0x7fbde81425f0 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdefe3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdefe3e640 10 monclient(hunting): authenticate will time out at 888.701848s 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbded3d9640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fbde81ccb70 0x7fbde813adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54370/0 (socket says 172.21.15.177:54370) 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbded3d9640 1 -- 172.21.15.177:0/3525331404 learned_addr learned my addr 172.21.15.177:0/3525331404 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 client.?.objecter ms_handle_connect 0x7fbde81ccb70 2024-03-29T10:47:53.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650205111 0 0) 0x7fbde80acd00 con 0x7fbde81ccb70 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdcc004eb0 con 0x7fbde81ccb70 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 client.?.objecter ms_handle_connect 0x7fbde809ac40 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 client.?.objecter ms_handle_connect 0x7fbde81425f0 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4127687605 0 0) 0x7fbdcc004eb0 con 0x7fbde81ccb70 2024-03-29T10:47:53.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbde80acd00 con 0x7fbde81ccb70 2024-03-29T10:47:53.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbdd8001a20 con 0x7fbde81ccb70 2024-03-29T10:47:53.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:53.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): dump: 2024-03-29T10:47:53.465 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.465 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.465 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.465 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406577548 0 0) 0x7fbde81cd520 con 0x7fbde809ac40 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.467 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:47:53.467 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdcc004eb0 con 0x7fbde809ac40 2024-03-29T10:47:53.467 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241550007 0 0) 0x7fbde81cd000 con 0x7fbde81425f0 2024-03-29T10:47:53.467 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.467 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.467 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.467 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 10 monclient(hunting): my global_id is 4157 2024-03-29T10:47:53.467 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.463+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde81cd520 con 0x7fbde81425f0 2024-03-29T10:47:53.467 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 512934059 0 0) 0x7fbde80acd00 con 0x7fbde81ccb70 2024-03-29T10:47:53.468 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 >> v1:172.21.15.5:6790/0 conn(0x7fbde81425f0 legacy=0x7fbde81e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.468 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 >> v1:172.21.15.5:6789/0 conn(0x7fbde809ac40 legacy=0x7fbde81da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.468 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:53.468 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 1 monclient: found mon.b 2024-03-29T10:47:53.468 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:53.468 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.468 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde80b1af0 con 0x7fbde81ccb70 2024-03-29T10:47:53.468 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:53.468 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdefe3e640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:47:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdefe3e640 10 monclient: _renew_subs 2024-03-29T10:47:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdefe3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3525331404 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde81cc300 con 0x7fbde81ccb70 2024-03-29T10:47:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdefe3e640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdefe3e640 10 monclient: _renew_subs 2024-03-29T10:47:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdefe3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3525331404 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde81daf80 con 0x7fbde81ccb70 2024-03-29T10:47:53.469 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdefe3e640 1 librados: init done 2024-03-29T10:47:53.470 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdd8001d40 con 0x7fbde81ccb70 2024-03-29T10:47:53.470 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:53.470 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbdd8003a60 con 0x7fbde81ccb70 2024-03-29T10:47:53.470 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.470 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:53.470 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 10 monclient: dump: 2024-03-29T10:47:53.470 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.470 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.470 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.470 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbddffff640 4 set_mon_vals no callback set 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbdd8004010 con 0x7fbde81ccb70 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2441045270 0 0) 0x7fbdd80069d0 con 0x7fbde81ccb70 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 10 client.4150.objecter ms_dispatch 0x7fbde8000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:47:53.471 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:47:53.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:47:53.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.464+0000 7fbdde7fc640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T10:47:53.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.465+0000 7fbdefe3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.465+0000 7fbdefe3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.465+0000 7fbdefe3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.465+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3525331404 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd0005080 con 0x7fbde81ccb70 2024-03-29T10:47:53.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.467+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbdd80063e0 con 0x7fbde81ccb70 2024-03-29T10:47:53.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.467+0000 7fbdde7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.467+0000 7fbdde7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:53.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.539+0000 7fbdefe3e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f863ddfb-54f0-4042-8f43-5355132f26af", "id": 6}] 2024-03-29T10:47:53.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.539+0000 7fbdefe3e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f863ddfb-54f0-4042-8f43-5355132f26af", "id": 6}] 2024-03-29T10:47:53.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.539+0000 7fbdefe3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.539+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3525331404 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f863ddfb-54f0-4042-8f43-5355132f26af", "id": 6} v 0) v1 -- 0x7fbdd00058b0 con 0x7fbde81ccb70 2024-03-29T10:47:53.526 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.543+0000 7fbdde7fc640 1 -- 172.21.15.177:0/3525331404 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f863ddfb-54f0-4042-8f43-5355132f26af", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3340991703 0 1113811849) 0x7fbdd8006640 con 0x7fbde81ccb70 2024-03-29T10:47:53.526 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.543+0000 7fbdde7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f863ddfb-54f0-4042-8f43-5355132f26af", "id": 6}] 2024-03-29T10:47:53.526 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.543+0000 7fbdde7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:53.526 INFO:teuthology.orchestra.run.smithi177.stdout:6 2024-03-29T10:47:53.527 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 10 librados: watch_flush enter 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 10 librados: watch_flush exit 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 10 client.4150.objecter successfully canceled tick 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 10 monclient: shutdown 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3525331404 >> v1:172.21.15.177:6789/0 conn(0x7fbde81ccb70 legacy=0x7fbde813adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3525331404 shutdown_connections 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3525331404 >> 172.21.15.177:0/3525331404 conn(0x7fbde8066b00 msgr2=0x7fbde8066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3525331404 shutdown_connections 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 1 -- 172.21.15.177:0/3525331404 wait complete. 2024-03-29T10:47:53.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.544+0000 7fbdefe3e640 1 librados: shutdown 2024-03-29T10:47:53.537 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 59ef6382-f411-4a44-9e59-59ad158f625b 7 2024-03-29T10:47:53.593 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:53.610+0000 7f4f87a17040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:47:53.604 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:53.622+0000 7fc1aea42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:47:53.638 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.655+0000 7f6f3d2c8640 10 monclient: get_monmap_and_config 2024-03-29T10:47:53.638 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.655+0000 7f6f3d2c8640 10 monclient: build_initial_monmap 2024-03-29T10:47:53.638 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.655+0000 7f6f3d2c8640 10 monclient: monmap: 2024-03-29T10:47:53.638 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:53.638 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.638 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:53.657129+0000 2024-03-29T10:47:53.638 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:53.657129+0000 2024-03-29T10:47:53.638 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:53.639 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.639 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:53.639 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:53.639 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:53.639 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.647 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 1 Processor -- start 2024-03-29T10:47:53.647 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 1 -- start start 2024-03-29T10:47:53.647 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 10 monclient: init 2024-03-29T10:47:53.647 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:53.647 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:53.647 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 10 monclient(hunting): picked mon.noname-a con 0x7f6f3809ac40 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:53.647 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 10 monclient(hunting): picked mon.noname-c con 0x7f6f3809c870 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:53.647 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 10 monclient(hunting): picked mon.noname-b con 0x7f6f380b4360 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:53.647 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3809dec0 con 0x7f6f3809ac40 2024-03-29T10:47:53.647 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3809da30 con 0x7f6f3809c870 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380b39c0 con 0x7f6f380b4360 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f3d2c8640 10 monclient(hunting): authenticate will time out at 888.895472s 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.656+0000 7f6f367fc640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f6f3809c870 0x7f6f3813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54374/0 (socket says 172.21.15.177:54374) 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f367fc640 1 -- 172.21.15.177:0/854238791 learned_addr learned my addr 172.21.15.177:0/854238791 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445469068 0 0) 0x7f6f3809da30 con 0x7f6f3809c870 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f20003520 con 0x7f6f3809c870 2024-03-29T10:47:53.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010420657 0 0) 0x7f6f380b39c0 con 0x7f6f380b4360 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient(hunting): my global_id is 4160 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f3809da30 con 0x7f6f380b4360 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4168009380 0 0) 0x7f6f20003520 con 0x7f6f3809c870 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 >> v1:172.21.15.5:6790/0 conn(0x7f6f380b4360 legacy=0x7f6f3814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 >> v1:172.21.15.5:6789/0 conn(0x7f6f3809ac40 legacy=0x7f6f380b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 monclient: found mon.noname-c 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380b5160 con 0x7f6f3809c870 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:53.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6f2c001690 con 0x7f6f3809c870 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f3d2c8640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f3d2c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: dump: 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f3d2c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f2c001870 con 0x7f6f3809c870 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f3d2c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f3d2c8640 4 set_mon_vals no callback set 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 1 -- 172.21.15.177:0/854238791 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6f2c001c80 con 0x7f6f3809c870 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.657+0000 7f6f35ffb640 10 monclient: dump: 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.658+0000 7f6f3d2c8640 10 monclient: shutdown 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.658+0000 7f6f3d2c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.658+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/854238791 >> v1:172.21.15.177:6789/0 conn(0x7f6f3809c870 legacy=0x7f6f3813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.658+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/854238791 shutdown_connections 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.658+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/854238791 >> 172.21.15.177:0/854238791 conn(0x7f6f38066b00 msgr2=0x7f6f38066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.658+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/854238791 shutdown_connections 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.658+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/854238791 wait complete. 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.658+0000 7f6f3d2c8640 10 monclient: build_initial_monmap 2024-03-29T10:47:53.652 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.658+0000 7f6f3d2c8640 10 monclient: monmap: 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 1 librados: starting msgr at 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 1 librados: starting objecter 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 1 Processor -- start 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 1 -- start start 2024-03-29T10:47:53.653 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 1 librados: setting wanted keys 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 1 librados: calling monclient init 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 10 monclient: init 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 10 monclient(hunting): picked mon.noname-b con 0x7f6f3809ac40 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 10 monclient(hunting): picked mon.noname-c con 0x7f6f380b74d0 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 10 monclient(hunting): picked mon.noname-a con 0x7f6f380b0f40 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f38142150 con 0x7f6f380b0f40 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3813a7f0 con 0x7f6f3809ac40 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380b7ab0 con 0x7f6f380b74d0 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f3d2c8640 10 monclient(hunting): authenticate will time out at 888.898536s 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f36ffd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f6f3809ac40 0x7f6f38140080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:54386/0 (socket says 172.21.15.177:54386) 2024-03-29T10:47:53.654 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.659+0000 7f6f36ffd640 1 -- 172.21.15.177:0/2945099278 learned_addr learned my addr 172.21.15.177:0/2945099278 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 client.?.objecter ms_handle_connect 0x7f6f3809ac40 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 client.?.objecter ms_handle_connect 0x7f6f380b74d0 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567656888 0 0) 0x7f6f3813a7f0 con 0x7f6f3809ac40 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): my global_id is 4156 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f24004eb0 con 0x7f6f3809ac40 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585550588 0 0) 0x7f6f380b7ab0 con 0x7f6f380b74d0 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.655 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): my global_id is 4163 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f3813a7f0 con 0x7f6f380b74d0 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898680206 0 0) 0x7f6f24004eb0 con 0x7f6f3809ac40 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f380b7ab0 con 0x7f6f3809ac40 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6f28001890 con 0x7f6f3809ac40 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): dump: 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.656 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 client.?.objecter ms_handle_connect 0x7f6f380b0f40 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3281315450 0 0) 0x7f6f3813a7f0 con 0x7f6f380b74d0 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f24004eb0 con 0x7f6f380b74d0 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6f2c001b80 con 0x7f6f380b74d0 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient(hunting): dump: 2024-03-29T10:47:53.657 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3444835353 0 0) 0x7f6f380b7ab0 con 0x7f6f3809ac40 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 >> v1:172.21.15.5:6790/0 conn(0x7f6f380b74d0 legacy=0x7f6f381399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 >> v1:172.21.15.5:6789/0 conn(0x7f6f380b0f40 legacy=0x7f6f3813a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:53.658 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 monclient: found mon.b 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f381c9cc0 con 0x7f6f3809ac40 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f3d2c8640 5 monclient: authenticate success, global_id 4156 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f3d2c8640 10 monclient: _renew_subs 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f3d2c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/2945099278 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f381c9f10 con 0x7f6f3809ac40 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f3d2c8640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f3d2c8640 10 monclient: _renew_subs 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f3d2c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.659 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/2945099278 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f38140970 con 0x7f6f3809ac40 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f3d2c8640 1 librados: init done 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f28001a70 con 0x7f6f3809ac40 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6f280037f0 con 0x7f6f3809ac40 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 monclient: dump: 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.660 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f357fa640 4 set_mon_vals no callback set 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6f28003d60 con 0x7f6f3809ac40 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2970083855 0 0) 0x7f6f28004010 con 0x7f6f3809ac40 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 10 client.4156.objecter ms_dispatch 0x7f6f38000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.660+0000 7f6f1f7fe640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.661+0000 7f6f3d2c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.661+0000 7f6f3d2c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.661+0000 7f6f3d2c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.661+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/2945099278 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f38140c10 con 0x7f6f3809ac40 2024-03-29T10:47:53.661 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.663+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6f28004380 con 0x7f6f3809ac40 2024-03-29T10:47:53.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.663+0000 7f6f1f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.663+0000 7f6f1f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:53.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.753+0000 7f6f3d2c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "59ef6382-f411-4a44-9e59-59ad158f625b", "id": 7}] 2024-03-29T10:47:53.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.753+0000 7f6f3d2c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "59ef6382-f411-4a44-9e59-59ad158f625b", "id": 7}] 2024-03-29T10:47:53.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.753+0000 7f6f3d2c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:47:53.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.753+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/2945099278 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd new", "uuid": "59ef6382-f411-4a44-9e59-59ad158f625b", "id": 7} v 0) v1 -- 0x7f6f38141e10 con 0x7f6f3809ac40 2024-03-29T10:47:53.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.756+0000 7f6f1f7fe640 1 -- 172.21.15.177:0/2945099278 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "59ef6382-f411-4a44-9e59-59ad158f625b", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2784893499 0 1371665406) 0x7f6f28005890 con 0x7f6f3809ac40 2024-03-29T10:47:53.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.756+0000 7f6f1f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "59ef6382-f411-4a44-9e59-59ad158f625b", "id": 7}] 2024-03-29T10:47:53.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.756+0000 7f6f1f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:53.739 INFO:teuthology.orchestra.run.smithi177.stdout:7 2024-03-29T10:47:53.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 10 librados: watch_flush enter 2024-03-29T10:47:53.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 10 librados: watch_flush exit 2024-03-29T10:47:53.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-29T10:47:53.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 10 client.4156.objecter successfully canceled tick 2024-03-29T10:47:53.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 10 monclient: shutdown 2024-03-29T10:47:53.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:53.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/2945099278 >> v1:172.21.15.177:6789/0 conn(0x7f6f3809ac40 legacy=0x7f6f38140080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/2945099278 shutdown_connections 2024-03-29T10:47:53.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/2945099278 >> 172.21.15.177:0/2945099278 conn(0x7f6f38066b00 msgr2=0x7f6f3813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:53.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/2945099278 shutdown_connections 2024-03-29T10:47:53.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 1 -- 172.21.15.177:0/2945099278 wait complete. 2024-03-29T10:47:53.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-29T10:47:53.757+0000 7f6f3d2c8640 1 librados: shutdown 2024-03-29T10:47:53.745 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:53.763+0000 7fc1aea42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:47:53.747 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:53.764+0000 7f4f87a17040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:47:53.753 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:47:53.754 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:47:53.756 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:47:53.756 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:47:53.756 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:47:53.758 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:47:53.758 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:47:53.759 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:47:53.760 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:47:53.761 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:47:53.761 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:47:53.762 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:47:53.763 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:47:53.763 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:47:53.765 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:47:53.765 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:47:53.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:47:53.767 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:47:53.767 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:47:53.767 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:47:53.770 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:47:53.770 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:47:53.770 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:47:53.772 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:47:53.772 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:53.877 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:53.894+0000 7f4f87a17040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:47:53.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.905+0000 7f0f384b3640 10 monclient: get_monmap_and_config 2024-03-29T10:47:53.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.905+0000 7f0f384b3640 10 monclient: build_initial_monmap 2024-03-29T10:47:53.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.905+0000 7f0f384b3640 10 monclient: monmap: 2024-03-29T10:47:53.888 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-03-29T10:47:53.888 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.888 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:53.906709+0000 2024-03-29T10:47:53.888 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:53.906709+0000 2024-03-29T10:47:53.888 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:53.888 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:53.888 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:53.888 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:53.888 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:53.889 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:53.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.906+0000 7f0f384b3640 1 Processor -- start 2024-03-29T10:47:53.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.906+0000 7f0f384b3640 1 -- start start 2024-03-29T10:47:53.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.906+0000 7f0f384b3640 10 monclient: init 2024-03-29T10:47:53.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.906+0000 7f0f384b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:53.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.906+0000 7f0f384b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:53.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f384b3640 10 monclient(hunting): picked mon.noname-a con 0x7f0f300b8240 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:53.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f384b3640 10 monclient(hunting): picked mon.noname-c con 0x7f0f300b1f60 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:53.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f384b3640 10 monclient(hunting): picked mon.noname-b con 0x7f0f3009c8c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f384b3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3009bd80 con 0x7f0f300b8240 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3624f640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0f300b8240 0x7f0f3013f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60072/0 (socket says 172.21.15.5:60072) 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3624f640 1 -- 172.21.15.5:0/1789254651 learned_addr learned my addr 172.21.15.5:0/1789254651 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f36a50640 1 --1- 172.21.15.5:0/1789254651 >> v1:172.21.15.5:6790/0 conn(0x7f0f3009c8c0 0x7f0f3014d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:51032/0 (socket says 172.21.15.5:51032) 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f384b3640 1 -- 172.21.15.5:0/1789254651 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3009b960 con 0x7f0f300b1f60 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f384b3640 1 -- 172.21.15.5:0/1789254651 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f300b2ec0 con 0x7f0f3009c8c0 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f384b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f384b3640 10 monclient(hunting): authenticate will time out at 887.148383s 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814071620 0 0) 0x7f0f3009bd80 con 0x7f0f300b8240 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:47:53.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f1c003520 con 0x7f0f300b8240 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512778488 0 0) 0x7f0f300b2ec0 con 0x7f0f3009c8c0 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3009bd80 con 0x7f0f3009c8c0 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447327193 0 0) 0x7f0f3009b960 con 0x7f0f300b1f60 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f300b2ec0 con 0x7f0f300b1f60 2024-03-29T10:47:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425141723 0 0) 0x7f0f1c003520 con 0x7f0f300b8240 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 >> v1:172.21.15.5:6790/0 conn(0x7f0f3009c8c0 legacy=0x7f0f3014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 >> v1:172.21.15.177:6789/0 conn(0x7f0f300b1f60 legacy=0x7f0f3009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: _finish_hunting 0 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 monclient: found mon.noname-a 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3013fb60 con 0x7f0f300b8240 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: _finish_auth 0 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: _check_auth_tickets 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f0f2c001690 con 0x7f0f300b8240 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: dump: 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f384b3640 5 monclient: authenticate success, global_id 4188 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f384b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f2c0018e0 con 0x7f0f300b8240 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 1 -- 172.21.15.5:0/1789254651 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f0f2c001cb0 con 0x7f0f300b8240 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.907+0000 7f0f3524d640 10 monclient: dump: 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.908+0000 7f0f384b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.908+0000 7f0f384b3640 4 set_mon_vals no callback set 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.908+0000 7f0f384b3640 10 monclient: shutdown 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.908+0000 7f0f384b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.908+0000 7f0f384b3640 1 -- 172.21.15.5:0/1789254651 >> v1:172.21.15.5:6789/0 conn(0x7f0f300b8240 legacy=0x7f0f3013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.908+0000 7f0f384b3640 1 -- 172.21.15.5:0/1789254651 shutdown_connections 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.908+0000 7f0f384b3640 1 -- 172.21.15.5:0/1789254651 >> 172.21.15.5:0/1789254651 conn(0x7f0f30066b00 msgr2=0x7f0f30066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.908+0000 7f0f384b3640 1 -- 172.21.15.5:0/1789254651 shutdown_connections 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.908+0000 7f0f384b3640 1 -- 172.21.15.5:0/1789254651 wait complete. 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.909+0000 7f0f384b3640 10 monclient: build_initial_monmap 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.909+0000 7f0f384b3640 10 monclient: monmap: 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-03-29T10:47:53.906 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.909+0000 7f0f384b3640 1 librados: starting msgr at 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.909+0000 7f0f384b3640 1 librados: starting objecter 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.909+0000 7f0f384b3640 1 Processor -- start 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 1 -- start start 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 1 librados: setting wanted keys 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 1 librados: calling monclient init 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 10 monclient: init 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:53.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 10 monclient(hunting): picked mon.noname-c con 0x7f0f3009c8c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:53.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 10 monclient(hunting): picked mon.noname-a con 0x7f0f300b8240 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:53.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 10 monclient(hunting): picked mon.noname-b con 0x7f0f30135ea0 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:53.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f301da670 con 0x7f0f300b8240 2024-03-29T10:47:53.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f30135830 con 0x7f0f30135ea0 2024-03-29T10:47:53.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f301db4f0 con 0x7f0f3009c8c0 2024-03-29T10:47:53.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:53.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f3624f640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f0f3009c8c0 0x7f0f301dade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:51042/0 (socket says 172.21.15.5:51042) 2024-03-29T10:47:53.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f384b3640 10 monclient(hunting): authenticate will time out at 887.151803s 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f3624f640 1 -- 172.21.15.5:0/1526376234 learned_addr learned my addr 172.21.15.5:0/1526376234 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f26ffd640 10 client.?.objecter ms_handle_connect 0x7f0f300b8240 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f26ffd640 10 client.?.objecter ms_handle_connect 0x7f0f3009c8c0 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.910+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518906862 0 0) 0x7f0f301da670 con 0x7f0f300b8240 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f14004eb0 con 0x7f0f300b8240 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908577487 0 0) 0x7f0f301db4f0 con 0x7f0f3009c8c0 2024-03-29T10:47:53.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): my global_id is 4190 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f301da670 con 0x7f0f3009c8c0 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 client.?.objecter ms_handle_connect 0x7f0f30135ea0 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 770169848 0 0) 0x7f0f14004eb0 con 0x7f0f300b8240 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f301db4f0 con 0x7f0f300b8240 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f0f200018f0 con 0x7f0f300b8240 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): dump: 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.911 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1946277104 0 0) 0x7f0f301da670 con 0x7f0f3009c8c0 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f14004eb0 con 0x7f0f3009c8c0 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f0f2c001a50 con 0x7f0f3009c8c0 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): dump: 2024-03-29T10:47:53.912 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495371067 0 0) 0x7f0f30135830 con 0x7f0f30135ea0 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:47:53.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f301da670 con 0x7f0f30135ea0 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 148450654 0 0) 0x7f0f301db4f0 con 0x7f0f300b8240 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 >> v1:172.21.15.5:6790/0 conn(0x7f0f3009c8c0 legacy=0x7f0f301dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 >> v1:172.21.15.177:6789/0 conn(0x7f0f30135ea0 legacy=0x7f0f30135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 monclient: found mon.a 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f30137f00 con 0x7f0f300b8240 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient: _finish_auth 0 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f384b3640 5 monclient: authenticate success, global_id 4191 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f384b3640 10 monclient: _renew_subs 2024-03-29T10:47:53.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f384b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f384b3640 1 -- 172.21.15.5:0/1526376234 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f301381e0 con 0x7f0f300b8240 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f384b3640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f384b3640 10 monclient: _renew_subs 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f384b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f384b3640 1 -- 172.21.15.5:0/1526376234 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f30143d50 con 0x7f0f300b8240 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f384b3640 1 librados: init done 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f20003610 con 0x7f0f300b8240 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f0f20003920 con 0x7f0f300b8240 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 monclient: dump: 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f34a4c640 4 set_mon_vals no callback set 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0f20003e10 con 0x7f0f300b8240 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f0f20005720 con 0x7f0f300b8240 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 10 client.4191.objecter ms_dispatch 0x7f0f30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.911+0000 7f0f26ffd640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T10:47:53.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.912+0000 7f0f384b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.912+0000 7f0f384b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.912+0000 7f0f384b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:47:53.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.912+0000 7f0f384b3640 1 -- 172.21.15.5:0/1526376234 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f18005080 con 0x7f0f300b8240 2024-03-29T10:47:53.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.914+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0f20005160 con 0x7f0f300b8240 2024-03-29T10:47:53.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.914+0000 7f0f26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:53.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.914+0000 7f0f26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:53.917 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:53.921+0000 7fc1aea42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:47:53.923 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:53.941+0000 7f4f87a17040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:47:53.950 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:53.968+0000 7fc1aea42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:47:53.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.976+0000 7f0f384b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:53.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.977+0000 7f0f384b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:53.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.977+0000 7f0f384b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:47:53.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.977+0000 7f0f384b3640 1 -- 172.21.15.5:0/1526376234 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f18005330 con 0x7f0f300b8240 2024-03-29T10:47:53.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.977+0000 7f0f26ffd640 1 -- 172.21.15.5:0/1526376234 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f0f200053b0 con 0x7f0f300b8240 2024-03-29T10:47:53.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.977+0000 7f0f26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:53.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.977+0000 7f0f26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:53.960 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-29T10:47:53.960 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:53.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 10 librados: watch_flush enter 2024-03-29T10:47:53.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 10 librados: watch_flush exit 2024-03-29T10:47:53.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T10:47:53.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 10 client.4191.objecter successfully canceled tick 2024-03-29T10:47:53.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 10 monclient: shutdown 2024-03-29T10:47:53.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:53.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 1 -- 172.21.15.5:0/1526376234 >> v1:172.21.15.5:6789/0 conn(0x7f0f300b8240 legacy=0x7f0f30134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:53.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 1 -- 172.21.15.5:0/1526376234 shutdown_connections 2024-03-29T10:47:53.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 1 -- 172.21.15.5:0/1526376234 >> 172.21.15.5:0/1526376234 conn(0x7f0f30066b00 msgr2=0x7f0f30069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:53.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 1 -- 172.21.15.5:0/1526376234 shutdown_connections 2024-03-29T10:47:53.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 1 -- 172.21.15.5:0/1526376234 wait complete. 2024-03-29T10:47:53.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-29T10:47:53.978+0000 7f0f384b3640 1 librados: shutdown 2024-03-29T10:47:53.970 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:47:53.971 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:47:54.035 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:54.053+0000 7f4f87a17040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:47:54.068 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:54.086+0000 7fc1aea42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:47:54.131 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:54.148+0000 7f4f87a17040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:47:54.167 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:54.184+0000 7fc1aea42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:47:54.284 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:54.302+0000 7f4f87a17040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:47:54.326 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:54.344+0000 7f4f87a17040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:47:54.327 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:54.345+0000 7fc1aea42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:47:54.364 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-03-29T10:47:54.381+0000 7fdf7bd49640 -1 Falling back to public interface 2024-03-29T10:47:54.370 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-03-29T10:47:54.387+0000 7f7d7470d640 -1 Falling back to public interface 2024-03-29T10:47:54.371 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:54.389+0000 7fc1aea42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:47:54.379 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-03-29T10:47:54.396+0000 7f254e1aa640 -1 Falling back to public interface 2024-03-29T10:47:54.385 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-03-29T10:47:54.403+0000 7f46f460f640 -1 Falling back to public interface 2024-03-29T10:47:54.387 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-03-29T10:47:54.404+0000 7f2a45d19640 -1 Falling back to public interface 2024-03-29T10:47:54.387 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-03-29T10:47:54.405+0000 7fe8f74a4640 -1 Falling back to public interface 2024-03-29T10:47:54.388 INFO:tasks.ceph.osd.3.smithi005.stderr:2024-03-29T10:47:54.406+0000 7ff76389d640 -1 Falling back to public interface 2024-03-29T10:47:54.389 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-03-29T10:47:54.406+0000 7f218d213640 -1 Falling back to public interface 2024-03-29T10:47:54.484 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:54.502+0000 7f4f87a17040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:47:54.535 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:54.553+0000 7fc1aea42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:47:54.776 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:54.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.883+0000 7ff56b124640 10 monclient: get_monmap_and_config 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.883+0000 7ff56b124640 10 monclient: build_initial_monmap 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.883+0000 7ff56b124640 10 monclient: monmap: 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:54.884547+0000 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:54.884547+0000 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:47:54.866 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:47:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 1 Processor -- start 2024-03-29T10:47:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 1 -- start start 2024-03-29T10:47:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 10 monclient: init 2024-03-29T10:47:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 10 monclient(hunting): picked mon.noname-c con 0x7ff5641457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 10 monclient(hunting): picked mon.noname-a con 0x7ff56413f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 10 monclient(hunting): picked mon.noname-b con 0x7ff5640a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640aaef0 con 0x7ff56413f650 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564145bd0 con 0x7ff5641457f0 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564145eb0 con 0x7ff5640a9d70 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.884+0000 7ff56b124640 10 monclient(hunting): authenticate will time out at 888.125966s 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5696c1640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7ff5640a9d70 0x7ff56414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:51058/0 (socket says 172.21.15.5:51058) 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5696c1640 1 -- 172.21.15.5:0/2503883006 learned_addr learned my addr 172.21.15.5:0/2503883006 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff563fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff56413f650 0x7ff5640b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60102/0 (socket says 172.21.15.5:60102) 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670477082 0 0) 0x7ff564145eb0 con 0x7ff5640a9d70 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient(hunting): my global_id is 4193 2024-03-29T10:47:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff554003520 con 0x7ff5640a9d70 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941868252 0 0) 0x7ff5640aaef0 con 0x7ff56413f650 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff564145eb0 con 0x7ff56413f650 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 640871749 0 0) 0x7ff554003520 con 0x7ff5640a9d70 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 >> v1:172.21.15.177:6789/0 conn(0x7ff5641457f0 legacy=0x7ff56413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 >> v1:172.21.15.5:6789/0 conn(0x7ff56413f650 legacy=0x7ff5640b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 monclient: found mon.noname-b 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:47:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff564146340 con 0x7ff5640a9d70 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff55c001940 con 0x7ff5640a9d70 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff56b124640 5 monclient: authenticate success, global_id 4193 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff56b124640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: dump: 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.877 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff56b124640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff55c001b20 con 0x7ff5640a9d70 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff56b124640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 1 -- 172.21.15.5:0/2503883006 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff55c001f70 con 0x7ff5640a9d70 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff56b124640 4 set_mon_vals no callback set 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.885+0000 7ff5637fe640 10 monclient: dump: 2024-03-29T10:47:54.878 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.886+0000 7ff56b124640 10 monclient: shutdown 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.886+0000 7ff56b124640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.886+0000 7ff56b124640 1 -- 172.21.15.5:0/2503883006 >> v1:172.21.15.5:6790/0 conn(0x7ff5640a9d70 legacy=0x7ff56414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.886+0000 7ff56b124640 1 -- 172.21.15.5:0/2503883006 shutdown_connections 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.886+0000 7ff56b124640 1 -- 172.21.15.5:0/2503883006 >> 172.21.15.5:0/2503883006 conn(0x7ff564066a00 msgr2=0x7ff564066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:54.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.886+0000 7ff56b124640 1 -- 172.21.15.5:0/2503883006 shutdown_connections 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.886+0000 7ff56b124640 1 -- 172.21.15.5:0/2503883006 wait complete. 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.886+0000 7ff56b124640 10 monclient: build_initial_monmap 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.886+0000 7ff56b124640 10 monclient: monmap: 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:47:54.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 1 librados: starting msgr at 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 1 librados: starting objecter 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 1 Processor -- start 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 1 -- start start 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 1 librados: setting wanted keys 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 1 librados: calling monclient init 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 10 monclient: init 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 10 monclient(hunting): picked mon.noname-b con 0x7ff5640a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 10 monclient(hunting): picked mon.noname-a con 0x7ff56413f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 10 monclient(hunting): picked mon.noname-c con 0x7ff5641457f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640ac280 con 0x7ff56413f650 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564142dc0 con 0x7ff5640a9d70 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640ac4d0 con 0x7ff5641457f0 2024-03-29T10:47:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff56b124640 10 monclient(hunting): authenticate will time out at 888.128995s 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff563fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff56413f650 0x7ff564136f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60104/0 (socket says 172.21.15.5:60104) 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.887+0000 7ff563fff640 1 -- 172.21.15.5:0/4195466619 learned_addr learned my addr 172.21.15.5:0/4195466619 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 client.?.objecter ms_handle_connect 0x7ff56413f650 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 client.?.objecter ms_handle_connect 0x7ff5641457f0 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 client.?.objecter ms_handle_connect 0x7ff5640a9d70 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363892624 0 0) 0x7ff5640ac280 con 0x7ff56413f650 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): my global_id is 4197 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff548004eb0 con 0x7ff56413f650 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083611481 0 0) 0x7ff5640ac4d0 con 0x7ff5641457f0 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): my global_id is 4196 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5640ac280 con 0x7ff5641457f0 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856199550 0 0) 0x7ff564142dc0 con 0x7ff5640a9d70 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5640ac4d0 con 0x7ff5640a9d70 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 101991153 0 0) 0x7ff548004eb0 con 0x7ff56413f650 2024-03-29T10:47:54.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff564142dc0 con 0x7ff56413f650 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2840364783 0 0) 0x7ff5640ac280 con 0x7ff5641457f0 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff548004eb0 con 0x7ff5641457f0 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff550001a50 con 0x7ff56413f650 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): dump: 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:54.884 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff55c001e30 con 0x7ff5641457f0 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): dump: 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:54.885 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 502942784 0 0) 0x7ff5640ac4d0 con 0x7ff5640a9d70 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5640ac280 con 0x7ff5640a9d70 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff558001730 con 0x7ff5640a9d70 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient(hunting): dump: 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:54.886 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2396248812 0 0) 0x7ff548004eb0 con 0x7ff5641457f0 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 >> v1:172.21.15.177:6789/0 conn(0x7ff5640a9d70 legacy=0x7ff5641426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 >> v1:172.21.15.5:6789/0 conn(0x7ff56413f650 legacy=0x7ff564136f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 monclient: found mon.c 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff564142140 con 0x7ff5641457f0 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:54.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff56b124640 5 monclient: authenticate success, global_id 4196 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff56b124640 10 monclient: _renew_subs 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff56b124640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff56b124640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5640ab620 con 0x7ff5641457f0 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff56b124640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff56b124640 10 monclient: _renew_subs 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff56b124640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff56b124640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff564141d20 con 0x7ff5641457f0 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff56b124640 1 librados: init done 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff55c001f90 con 0x7ff5641457f0 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff562ffd640 4 set_mon_vals no callback set 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff55c004e10 con 0x7ff5641457f0 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:54.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.888+0000 7ff5617fa640 10 monclient: dump: 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.889+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff55c005380 con 0x7ff5641457f0 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.889+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7ff55c006bb0 con 0x7ff5641457f0 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.889+0000 7ff5617fa640 10 client.4196.objecter ms_dispatch 0x7ff564000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:54.889 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.889+0000 7ff5617fa640 3 client.4196.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:54.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.889+0000 7ff5617fa640 3 client.4196.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:54.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.889+0000 7ff5617fa640 20 client.4196.objecter dump_active .. 0 homeless 2024-03-29T10:47:54.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.889+0000 7ff56b124640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:54.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.889+0000 7ff56b124640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:54.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.889+0000 7ff56b124640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:47:54.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.889+0000 7ff56b124640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff544005080 con 0x7ff5641457f0 2024-03-29T10:47:54.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.891+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff55c006590 con 0x7ff5641457f0 2024-03-29T10:47:54.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.891+0000 7ff5617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:54.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.891+0000 7ff5617fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:54.935 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.953+0000 7ff56b124640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:54.936 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.953+0000 7ff56b124640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:54.936 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.953+0000 7ff56b124640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:47:54.936 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.953+0000 7ff56b124640 1 -- 172.21.15.5:0/4195466619 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff544005330 con 0x7ff5641457f0 2024-03-29T10:47:54.936 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.954+0000 7ff5617fa640 1 -- 172.21.15.5:0/4195466619 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7ff55c0067e0 con 0x7ff5641457f0 2024-03-29T10:47:54.936 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.954+0000 7ff5617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:54.936 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.954+0000 7ff5617fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:54.937 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:47:54.937 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:54.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 10 librados: watch_flush enter 2024-03-29T10:47:54.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 10 librados: watch_flush exit 2024-03-29T10:47:54.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 20 client.4196.objecter shutdown clearing up homeless session... 2024-03-29T10:47:54.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 10 client.4196.objecter successfully canceled tick 2024-03-29T10:47:54.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 10 monclient: shutdown 2024-03-29T10:47:54.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:54.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 1 -- 172.21.15.5:0/4195466619 >> v1:172.21.15.5:6790/0 conn(0x7ff5641457f0 legacy=0x7ff5641e2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:54.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 1 -- 172.21.15.5:0/4195466619 shutdown_connections 2024-03-29T10:47:54.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 1 -- 172.21.15.5:0/4195466619 >> 172.21.15.5:0/4195466619 conn(0x7ff564066a00 msgr2=0x7ff564066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:54.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 1 -- 172.21.15.5:0/4195466619 shutdown_connections 2024-03-29T10:47:54.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 1 -- 172.21.15.5:0/4195466619 wait complete. 2024-03-29T10:47:54.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:47:54.955+0000 7ff56b124640 1 librados: shutdown 2024-03-29T10:47:54.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:55.371 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:55.389+0000 7fc1aea42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:47:55.597 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:55.615+0000 7f4f87a17040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:47:55.659 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:55.676+0000 7fc1aea42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:47:55.712 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:55.730+0000 7fc1aea42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:47:55.832 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:55.850+0000 7fc1aea42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:47:55.874 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:55.892+0000 7f4f87a17040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:47:55.925 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:55.943+0000 7f4f87a17040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:47:56.041 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-29T10:47:56.059+0000 7f4f87a17040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:47:56.846 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:56.863+0000 7fc14fedd640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:47:56.847 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-29T10:47:56.864+0000 7fc14fedd640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:47:57.804 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-03-29T10:47:57.822+0000 7fdf7bd49640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:47:57.826 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-03-29T10:47:57.845+0000 7f7d7470d640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:47:57.866 INFO:tasks.ceph.osd.3.smithi005.stderr:2024-03-29T10:47:57.884+0000 7ff76389d640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:47:57.895 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-03-29T10:47:57.913+0000 7f254e1aa640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:47:57.898 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-03-29T10:47:57.916+0000 7f218d213640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:47:57.899 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-03-29T10:47:57.918+0000 7fe8f74a4640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:47:57.905 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-03-29T10:47:57.923+0000 7f2a45d19640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:47:57.932 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-03-29T10:47:57.949+0000 7f46f460f640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:48:01.755 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.924+0000 7f7516ac1640 10 monclient: get_monmap_and_config 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.924+0000 7f7516ac1640 10 monclient: build_initial_monmap 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.924+0000 7f7516ac1640 10 monclient: monmap: 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:48:01.926712+0000 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:48:01.926712+0000 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:48:01.908 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:48:01.909 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:01.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.925+0000 7f7516ac1640 1 Processor -- start 2024-03-29T10:48:01.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.925+0000 7f7516ac1640 1 -- start start 2024-03-29T10:48:01.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.925+0000 7f7516ac1640 10 monclient: init 2024-03-29T10:48:01.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.926+0000 7f7516ac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:01.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.926+0000 7f7516ac1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:01.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.926+0000 7f7516ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f75100b0770 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:01.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.926+0000 7f7516ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f751014a700 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:01.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.926+0000 7f7516ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f75100ac5a0 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:01.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.926+0000 7f7516ac1640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75100aba70 con 0x7f75100b0770 2024-03-29T10:48:01.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.926+0000 7f7516ac1640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75100aecb0 con 0x7f751014a700 2024-03-29T10:48:01.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.926+0000 7f7516ac1640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75100af150 con 0x7f75100ac5a0 2024-03-29T10:48:01.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.926+0000 7f7516ac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:01.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.926+0000 7f7516ac1640 10 monclient(hunting): authenticate will time out at 895.168770s 2024-03-29T10:48:01.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.927+0000 7f751505e640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f75100ac5a0 0x7f7510151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:44702/0 (socket says 172.21.15.5:44702) 2024-03-29T10:48:01.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.927+0000 7f751505e640 1 -- 172.21.15.5:0/1452159247 learned_addr learned my addr 172.21.15.5:0/1452159247 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:01.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.927+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097136002 0 0) 0x7f75100af150 con 0x7f75100ac5a0 2024-03-29T10:48:01.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.927+0000 7f750f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:01.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.927+0000 7f750f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:01.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.927+0000 7f750f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:01.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.927+0000 7f750f7fe640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:48:01.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.927+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7500003520 con 0x7f75100ac5a0 2024-03-29T10:48:01.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108247854 0 0) 0x7f75100aecb0 con 0x7f751014a700 2024-03-29T10:48:01.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:01.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:01.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:01.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:48:01.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75100af150 con 0x7f751014a700 2024-03-29T10:48:01.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37132438 0 0) 0x7f75100aba70 con 0x7f75100b0770 2024-03-29T10:48:01.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:01.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:01.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:01.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:48:01.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75100aecb0 con 0x7f75100b0770 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1756852302 0 0) 0x7f7500003520 con 0x7f75100ac5a0 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 >> v1:172.21.15.177:6789/0 conn(0x7f751014a700 legacy=0x7f75100abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 >> v1:172.21.15.5:6789/0 conn(0x7f75100b0770 legacy=0x7f7510149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75100ac9d0 con 0x7f75100ac5a0 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7508001a10 con 0x7f75100ac5a0 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:01.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f750f7fe640 10 monclient: dump: 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f7516ac1640 5 monclient: authenticate success, global_id 4238 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.928+0000 7f7516ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7508001b70 con 0x7f75100ac5a0 2024-03-29T10:48:01.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f750f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f750f7fe640 1 -- 172.21.15.5:0/1452159247 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7508001f80 con 0x7f75100ac5a0 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f750f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f750f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f750f7fe640 10 monclient: dump: 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:01.918 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f7516ac1640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f7516ac1640 4 set_mon_vals no callback set 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f7516ac1640 10 monclient: shutdown 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f7516ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.929+0000 7f7516ac1640 1 -- 172.21.15.5:0/1452159247 >> v1:172.21.15.5:6790/0 conn(0x7f75100ac5a0 legacy=0x7f7510151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.930+0000 7f7516ac1640 1 -- 172.21.15.5:0/1452159247 shutdown_connections 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.930+0000 7f7516ac1640 1 -- 172.21.15.5:0/1452159247 >> 172.21.15.5:0/1452159247 conn(0x7f751006b660 msgr2=0x7f751006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.930+0000 7f7516ac1640 1 -- 172.21.15.5:0/1452159247 shutdown_connections 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.930+0000 7f7516ac1640 1 -- 172.21.15.5:0/1452159247 wait complete. 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.931+0000 7f7516ac1640 10 monclient: build_initial_monmap 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.931+0000 7f7516ac1640 10 monclient: monmap: 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:48:01.919 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:48:01.922 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:01.922 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:01.922 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:01.922 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:48:01.922 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:48:01.923 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:01.937 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.931+0000 7f7516ac1640 1 librados: starting msgr at 2024-03-29T10:48:01.937 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.931+0000 7f7516ac1640 1 librados: starting objecter 2024-03-29T10:48:01.937 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.931+0000 7f7516ac1640 1 Processor -- start 2024-03-29T10:48:01.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.931+0000 7f7516ac1640 1 -- start start 2024-03-29T10:48:01.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.931+0000 7f7516ac1640 1 librados: setting wanted keys 2024-03-29T10:48:01.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.931+0000 7f7516ac1640 1 librados: calling monclient init 2024-03-29T10:48:01.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.931+0000 7f7516ac1640 10 monclient: init 2024-03-29T10:48:01.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.932+0000 7f7516ac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:01.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.932+0000 7f7516ac1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:01.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.932+0000 7f7516ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f75100ac5a0 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:01.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.932+0000 7f7516ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f75100b0770 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:01.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.932+0000 7f7516ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f751014a700 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:01.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.932+0000 7f7516ac1640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7510126590 con 0x7f751014a700 2024-03-29T10:48:01.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.932+0000 7f7516ac1640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751012b4d0 con 0x7f75100b0770 2024-03-29T10:48:01.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.933+0000 7f7516ac1640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7510129e10 con 0x7f75100ac5a0 2024-03-29T10:48:01.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.933+0000 7f7516ac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:01.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.933+0000 7f7516ac1640 10 monclient(hunting): authenticate will time out at 895.175108s 2024-03-29T10:48:01.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.933+0000 7f751485d640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f75100ac5a0 0x7f7510129070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:44848/0 (socket says 172.21.15.5:44848) 2024-03-29T10:48:01.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.933+0000 7f751485d640 1 -- 172.21.15.5:0/2473640502 learned_addr learned my addr 172.21.15.5:0/2473640502 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:01.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.933+0000 7f750d7fa640 10 client.?.objecter ms_handle_connect 0x7f75100ac5a0 2024-03-29T10:48:01.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.933+0000 7f750d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:01.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.933+0000 7f750d7fa640 10 client.?.objecter ms_handle_connect 0x7f751014a700 2024-03-29T10:48:01.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.933+0000 7f750d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:01.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.934+0000 7f750d7fa640 10 client.?.objecter ms_handle_connect 0x7f75100b0770 2024-03-29T10:48:01.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.934+0000 7f750d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:01.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.934+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78732917 0 0) 0x7f7510126590 con 0x7f751014a700 2024-03-29T10:48:01.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.934+0000 7f750d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:01.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.934+0000 7f750d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:01.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.934+0000 7f750d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:01.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.934+0000 7f750d7fa640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:48:01.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.934+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f4004eb0 con 0x7f751014a700 2024-03-29T10:48:01.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.934+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455339089 0 0) 0x7f751012b4d0 con 0x7f75100b0770 2024-03-29T10:48:01.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:01.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:01.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:01.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:48:01.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7510126590 con 0x7f75100b0770 2024-03-29T10:48:01.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1211981674 0 0) 0x7f74f4004eb0 con 0x7f751014a700 2024-03-29T10:48:01.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f751012b4d0 con 0x7f751014a700 2024-03-29T10:48:01.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7508001e80 con 0x7f751014a700 2024-03-29T10:48:01.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:01.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:01.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 10 monclient(hunting): dump: 2024-03-29T10:48:01.944 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:01.945 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:01.945 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:01.945 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:01.945 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:01.945 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:01.945 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:01.946 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:01.946 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:01.946 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:01.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.935+0000 7f750d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:01.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501106983 0 0) 0x7f7510129e10 con 0x7f75100ac5a0 2024-03-29T10:48:01.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:01.947 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:01.947 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:01.947 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:48:01.947 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f4004eb0 con 0x7f75100ac5a0 2024-03-29T10:48:01.947 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1158625070 0 0) 0x7f7510126590 con 0x7f75100b0770 2024-03-29T10:48:01.948 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7510129e10 con 0x7f75100b0770 2024-03-29T10:48:01.948 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f74fc001910 con 0x7f75100b0770 2024-03-29T10:48:01.948 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:01.948 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:01.948 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient(hunting): dump: 2024-03-29T10:48:01.948 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:01.949 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:01.949 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:01.949 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:01.949 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:01.949 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:01.949 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:01.950 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:01.950 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:01.950 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:01.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2875482478 0 0) 0x7f751012b4d0 con 0x7f751014a700 2024-03-29T10:48:01.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 >> v1:172.21.15.5:6790/0 conn(0x7f75100ac5a0 legacy=0x7f7510129070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:01.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 >> v1:172.21.15.177:6789/0 conn(0x7f75100b0770 legacy=0x7f75101dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:01.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:48:01.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 1 monclient: found mon.a 2024-03-29T10:48:01.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:01.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:48:01.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7510129fa0 con 0x7f751014a700 2024-03-29T10:48:01.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:48:01.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:48:01.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f750d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:01.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f7516ac1640 5 monclient: authenticate success, global_id 4242 2024-03-29T10:48:01.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f7516ac1640 10 monclient: _renew_subs 2024-03-29T10:48:01.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f7516ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:48:01.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f7516ac1640 1 -- 172.21.15.5:0/2473640502 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751012a250 con 0x7f751014a700 2024-03-29T10:48:01.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f7516ac1640 10 client.4242.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:01.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f7516ac1640 10 monclient: _renew_subs 2024-03-29T10:48:01.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f7516ac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:48:01.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f7516ac1640 1 -- 172.21.15.5:0/2473640502 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75101dc340 con 0x7f751014a700 2024-03-29T10:48:01.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.936+0000 7f7516ac1640 1 librados: init done 2024-03-29T10:48:01.954 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.938+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7508004e10 con 0x7f751014a700 2024-03-29T10:48:01.954 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.938+0000 7f750d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:01.954 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.938+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7508005120 con 0x7f751014a700 2024-03-29T10:48:01.954 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.938+0000 7f750d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:01.954 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.938+0000 7f750d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:01.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.938+0000 7f750d7fa640 10 monclient: dump: 2024-03-29T10:48:01.955 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:01.955 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:01.955 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:01.955 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:01.955 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:01.956 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:01.956 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:01.956 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:01.956 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:01.956 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:01.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.938+0000 7f750effd640 4 set_mon_vals no callback set 2024-03-29T10:48:01.957 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.938+0000 7f74faffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:01.957 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.939+0000 7f74faffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:01.957 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.939+0000 7f74faffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:48:01.957 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.939+0000 7f74faffd640 1 -- 172.21.15.5:0/2473640502 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75100aafa0 con 0x7f751014a700 2024-03-29T10:48:01.957 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.941+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f750801b0f0 con 0x7f751014a700 2024-03-29T10:48:01.957 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.942+0000 7f751485d640 10 monclient: get_auth_request con 0x7f74f406e020 auth_method 0 2024-03-29T10:48:01.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.942+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f7508084e10 con 0x7f751014a700 2024-03-29T10:48:01.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.942+0000 7f750d7fa640 10 client.4242.objecter ms_dispatch 0x7f7510000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:01.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.942+0000 7f750d7fa640 3 client.4242.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:01.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.942+0000 7f750d7fa640 3 client.4242.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:01.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.942+0000 7f750d7fa640 20 client.4242.objecter dump_active .. 0 homeless 2024-03-29T10:48:01.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.942+0000 7f750d7fa640 10 client.4242.objecter ms_handle_connect 0x7f74f406e020 2024-03-29T10:48:01.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.953+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7508050520 con 0x7f751014a700 2024-03-29T10:48:01.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.953+0000 7f750d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:01.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:01.953+0000 7f750d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:02.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.361+0000 7f74faffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:02.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.361+0000 7f74faffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:02.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.361+0000 7f74faffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:48:02.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.361+0000 7f74faffd640 1 -- 172.21.15.5:0/2473640502 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75100ab560 con 0x7f751014a700 2024-03-29T10:48:02.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.363+0000 7f750d7fa640 1 -- 172.21.15.5:0/2473640502 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f7508052620 con 0x7f751014a700 2024-03-29T10:48:02.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.363+0000 7f750d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:02.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.363+0000 7f750d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:02.346 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:48:02.346 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:02.349 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.366+0000 7f74faffd640 10 librados: watch_flush enter 2024-03-29T10:48:02.349 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.366+0000 7f74faffd640 10 librados: watch_flush exit 2024-03-29T10:48:02.349 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.366+0000 7f74faffd640 20 client.4242.objecter shutdown clearing up homeless session... 2024-03-29T10:48:02.349 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.366+0000 7f74faffd640 10 client.4242.objecter successfully canceled tick 2024-03-29T10:48:02.350 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.366+0000 7f74faffd640 1 -- 172.21.15.5:0/2473640502 >> v1:172.21.15.5:6816/2847500151 conn(0x7f74f406e020 legacy=0x7f74f40704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:02.350 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.367+0000 7f74faffd640 10 monclient: shutdown 2024-03-29T10:48:02.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.367+0000 7f74faffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:02.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.367+0000 7f74faffd640 1 -- 172.21.15.5:0/2473640502 >> v1:172.21.15.5:6789/0 conn(0x7f751014a700 legacy=0x7f75101e6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:02.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.367+0000 7f74faffd640 1 -- 172.21.15.5:0/2473640502 shutdown_connections 2024-03-29T10:48:02.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.367+0000 7f74faffd640 1 -- 172.21.15.5:0/2473640502 >> 172.21.15.5:0/2473640502 conn(0x7f751006b660 msgr2=0x7f7510149530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:02.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.367+0000 7f74faffd640 1 -- 172.21.15.5:0/2473640502 shutdown_connections 2024-03-29T10:48:02.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.367+0000 7f74faffd640 1 -- 172.21.15.5:0/2473640502 wait complete. 2024-03-29T10:48:02.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:02.367+0000 7f74faffd640 1 librados: shutdown 2024-03-29T10:48:02.366 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:09.173 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:09.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.343+0000 7f9f07a86640 10 monclient: get_monmap_and_config 2024-03-29T10:48:09.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.343+0000 7f9f07a86640 10 monclient: build_initial_monmap 2024-03-29T10:48:09.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.343+0000 7f9f07a86640 10 monclient: monmap: 2024-03-29T10:48:09.325 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:09.325 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:09.325 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:48:09.344097+0000 2024-03-29T10:48:09.326 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:48:09.344097+0000 2024-03-29T10:48:09.326 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:09.326 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:09.326 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:09.326 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:48:09.326 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:48:09.326 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:09.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.343+0000 7f9f07a86640 1 Processor -- start 2024-03-29T10:48:09.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.344+0000 7f9f07a86640 1 -- start start 2024-03-29T10:48:09.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.344+0000 7f9f07a86640 10 monclient: init 2024-03-29T10:48:09.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f07a86640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:09.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f07a86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:09.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f07a86640 10 monclient(hunting): picked mon.noname-c con 0x7f9f001457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:09.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f07a86640 10 monclient(hunting): picked mon.noname-b con 0x7f9f0013f650 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:09.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f07a86640 10 monclient(hunting): picked mon.noname-a con 0x7f9f000a9d70 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:09.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f07a86640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f000aaef0 con 0x7f9f000a9d70 2024-03-29T10:48:09.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f07a86640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f00145bd0 con 0x7f9f001457f0 2024-03-29T10:48:09.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f07a86640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f00145eb0 con 0x7f9f0013f650 2024-03-29T10:48:09.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f07a86640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:09.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f07a86640 10 monclient(hunting): authenticate will time out at 902.586263s 2024-03-29T10:48:09.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f05822640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f9f001457f0 0x7f9f0013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:58700/0 (socket says 172.21.15.5:58700) 2024-03-29T10:48:09.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f05822640 1 -- 172.21.15.5:0/2951550298 learned_addr learned my addr 172.21.15.5:0/2951550298 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:09.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137248237 0 0) 0x7f9f00145eb0 con 0x7f9f0013f650 2024-03-29T10:48:09.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f04820640 10 monclient(hunting): _negotiate 2024-03-29T10:48:09.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f04820640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:09.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f04820640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:09.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f04820640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:48:09.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.345+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eec003520 con 0x7f9f0013f650 2024-03-29T10:48:09.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701257822 0 0) 0x7f9f000aaef0 con 0x7f9f000a9d70 2024-03-29T10:48:09.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient(hunting): _negotiate 2024-03-29T10:48:09.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f00145eb0 con 0x7f9f000a9d70 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1584212018 0 0) 0x7f9eec003520 con 0x7f9f0013f650 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 >> v1:172.21.15.177:6789/0 conn(0x7f9f001457f0 legacy=0x7f9f0013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 >> v1:172.21.15.5:6789/0 conn(0x7f9f000a9d70 legacy=0x7f9f0014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: _finish_hunting 0 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 1 monclient: found mon.noname-b 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f00146340 con 0x7f9f0013f650 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: _finish_auth 0 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: _check_auth_tickets 2024-03-29T10:48:09.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:09.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f07a86640 5 monclient: authenticate success, global_id 4244 2024-03-29T10:48:09.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f07a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:09.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9ef4001690 con 0x7f9f0013f650 2024-03-29T10:48:09.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:09.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:09.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: dump: 2024-03-29T10:48:09.331 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:09.331 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:09.331 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:09.331 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f07a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ef4001870 con 0x7f9f0013f650 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 1 -- 172.21.15.5:0/2951550298 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9ef4001c40 con 0x7f9f0013f650 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:09.332 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f04820640 10 monclient: dump: 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f07a86640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:09.333 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.346+0000 7f9f07a86640 4 set_mon_vals no callback set 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.347+0000 7f9f07a86640 10 monclient: shutdown 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.347+0000 7f9f07a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.347+0000 7f9f07a86640 1 -- 172.21.15.5:0/2951550298 >> v1:172.21.15.5:6790/0 conn(0x7f9f0013f650 legacy=0x7f9f000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.347+0000 7f9f07a86640 1 -- 172.21.15.5:0/2951550298 shutdown_connections 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.347+0000 7f9f07a86640 1 -- 172.21.15.5:0/2951550298 >> 172.21.15.5:0/2951550298 conn(0x7f9f00066a00 msgr2=0x7f9f00066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.348+0000 7f9f07a86640 1 -- 172.21.15.5:0/2951550298 shutdown_connections 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.348+0000 7f9f07a86640 1 -- 172.21.15.5:0/2951550298 wait complete. 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.348+0000 7f9f07a86640 10 monclient: build_initial_monmap 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.348+0000 7f9f07a86640 10 monclient: monmap: 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:48:09.339 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.349+0000 7f9f07a86640 1 librados: starting msgr at 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.349+0000 7f9f07a86640 1 librados: starting objecter 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.349+0000 7f9f07a86640 1 Processor -- start 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.349+0000 7f9f07a86640 1 -- start start 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.349+0000 7f9f07a86640 1 librados: setting wanted keys 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.349+0000 7f9f07a86640 1 librados: calling monclient init 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.349+0000 7f9f07a86640 10 monclient: init 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.350+0000 7f9f07a86640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.350+0000 7f9f07a86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:09.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.350+0000 7f9f07a86640 10 monclient(hunting): picked mon.noname-b con 0x7f9f000a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.350+0000 7f9f07a86640 10 monclient(hunting): picked mon.noname-a con 0x7f9f0013f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.350+0000 7f9f07a86640 10 monclient(hunting): picked mon.noname-c con 0x7f9f001457f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.350+0000 7f9f07a86640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0006f620 con 0x7f9f0013f650 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.350+0000 7f9f07a86640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f001db9a0 con 0x7f9f000a9d70 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.350+0000 7f9f07a86640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0006f870 con 0x7f9f001457f0 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.350+0000 7f9f07a86640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.350+0000 7f9f07a86640 10 monclient(hunting): authenticate will time out at 902.591938s 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9f05021640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9f0013f650 0x7f9f001378a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38844/0 (socket says 172.21.15.5:38844) 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9f05021640 1 -- 172.21.15.5:0/3352343323 learned_addr learned my addr 172.21.15.5:0/3352343323 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 client.?.objecter ms_handle_connect 0x7f9f0013f650 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 client.?.objecter ms_handle_connect 0x7f9f000a9d70 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 client.?.objecter ms_handle_connect 0x7f9f001457f0 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:09.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284296193 0 0) 0x7f9f0006f870 con 0x7f9f001457f0 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee4004eb0 con 0x7f9f001457f0 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941116432 0 0) 0x7f9f0006f620 con 0x7f9f0013f650 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.351+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0006f870 con 0x7f9f0013f650 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482214205 0 0) 0x7f9f001db9a0 con 0x7f9f000a9d70 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:09.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0006f620 con 0x7f9f000a9d70 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2720917042 0 0) 0x7f9ee4004eb0 con 0x7f9f001457f0 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f001db9a0 con 0x7f9f001457f0 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9ef8001cb0 con 0x7f9f001457f0 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): dump: 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:09.343 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1991690238 0 0) 0x7f9f0006f620 con 0x7f9f000a9d70 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ee4004eb0 con 0x7f9f000a9d70 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9efc0017c0 con 0x7f9f000a9d70 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 10 monclient(hunting): dump: 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:09.344 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1610204055 0 0) 0x7f9f001db9a0 con 0x7f9f001457f0 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.352+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 >> v1:172.21.15.177:6789/0 conn(0x7f9f000a9d70 legacy=0x7f9f001dbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 >> v1:172.21.15.5:6789/0 conn(0x7f9f0013f650 legacy=0x7f9f001378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 1 monclient: found mon.c 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0006fb50 con 0x7f9f001457f0 2024-03-29T10:48:09.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9f07a86640 5 monclient: authenticate success, global_id 4247 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9f07a86640 10 monclient: _renew_subs 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9f07a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9f07a86640 1 -- 172.21.15.5:0/3352343323 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0006de90 con 0x7f9f001457f0 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9f07a86640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9f07a86640 10 monclient: _renew_subs 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9f07a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9f07a86640 1 -- 172.21.15.5:0/3352343323 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0006e6e0 con 0x7f9f001457f0 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9f07a86640 1 librados: init done 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ef8003990 con 0x7f9f001457f0 2024-03-29T10:48:09.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9ef8003d20 con 0x7f9f001457f0 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef27fc640 10 monclient: dump: 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:09.347 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.353+0000 7f9ef37fe640 4 set_mon_vals no callback set 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.355+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f9ef8003e80 con 0x7f9f001457f0 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.356+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f9ef8083b80 con 0x7f9f001457f0 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.356+0000 7f9ef27fc640 10 client.4247.objecter ms_dispatch 0x7f9f00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.356+0000 7f9ef27fc640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.356+0000 7f9ef27fc640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.356+0000 7f9f07a86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.356+0000 7f9f07a86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.356+0000 7f9f07a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.356+0000 7f9f07a86640 1 -- 172.21.15.5:0/3352343323 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee8005080 con 0x7f9f001457f0 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.356+0000 7f9ef27fc640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-29T10:48:09.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.356+0000 7f9f05822640 10 monclient: get_auth_request con 0x7f9ee406df90 auth_method 0 2024-03-29T10:48:09.349 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.357+0000 7f9ef27fc640 10 client.4247.objecter ms_handle_connect 0x7f9ee406df90 2024-03-29T10:48:09.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.369+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ef804f460 con 0x7f9f001457f0 2024-03-29T10:48:09.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.369+0000 7f9ef27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:09.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.369+0000 7f9ef27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:09.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.756+0000 7f9f07a86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:09.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.756+0000 7f9f07a86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:09.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.756+0000 7f9f07a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:09.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.756+0000 7f9f07a86640 1 -- 172.21.15.5:0/3352343323 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ee8005330 con 0x7f9f001457f0 2024-03-29T10:48:09.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.757+0000 7f9ef27fc640 1 -- 172.21.15.5:0/3352343323 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f9ef8051390 con 0x7f9f001457f0 2024-03-29T10:48:09.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.757+0000 7f9ef27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:09.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.757+0000 7f9ef27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:09.740 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:48:09.740 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:09.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 10 librados: watch_flush enter 2024-03-29T10:48:09.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 10 librados: watch_flush exit 2024-03-29T10:48:09.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-29T10:48:09.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 10 client.4247.objecter successfully canceled tick 2024-03-29T10:48:09.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 1 -- 172.21.15.5:0/3352343323 >> v1:172.21.15.5:6816/2847500151 conn(0x7f9ee406df90 legacy=0x7f9ee4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:09.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 10 monclient: shutdown 2024-03-29T10:48:09.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:09.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 1 -- 172.21.15.5:0/3352343323 >> v1:172.21.15.5:6790/0 conn(0x7f9f001457f0 legacy=0x7f9f001e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:09.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 1 -- 172.21.15.5:0/3352343323 shutdown_connections 2024-03-29T10:48:09.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 1 -- 172.21.15.5:0/3352343323 >> 172.21.15.5:0/3352343323 conn(0x7f9f00066a00 msgr2=0x7f9f0013dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:09.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 1 -- 172.21.15.5:0/3352343323 shutdown_connections 2024-03-29T10:48:09.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 1 -- 172.21.15.5:0/3352343323 wait complete. 2024-03-29T10:48:09.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:09.761+0000 7f9f07a86640 1 librados: shutdown 2024-03-29T10:48:09.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:16.565 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:16.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.722+0000 7febe1aed640 10 monclient: get_monmap_and_config 2024-03-29T10:48:16.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.722+0000 7febe1aed640 10 monclient: build_initial_monmap 2024-03-29T10:48:16.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.722+0000 7febe1aed640 10 monclient: monmap: 2024-03-29T10:48:16.704 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:16.705 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:16.705 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:48:16.723262+0000 2024-03-29T10:48:16.705 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:48:16.723262+0000 2024-03-29T10:48:16.705 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:16.705 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:16.705 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:16.705 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:48:16.705 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:48:16.705 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:16.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.723+0000 7febe1aed640 1 Processor -- start 2024-03-29T10:48:16.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.723+0000 7febe1aed640 1 -- start start 2024-03-29T10:48:16.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.723+0000 7febe1aed640 10 monclient: init 2024-03-29T10:48:16.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.724+0000 7febe1aed640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:16.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.724+0000 7febe1aed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:16.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.724+0000 7febe1aed640 10 monclient(hunting): picked mon.noname-a con 0x7febdc0b0770 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:16.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.724+0000 7febe1aed640 10 monclient(hunting): picked mon.noname-c con 0x7febdc14a700 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:16.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.724+0000 7febe1aed640 10 monclient(hunting): picked mon.noname-b con 0x7febdc0ac5a0 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:16.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febdbfff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7febdc0ac5a0 0x7febdc151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:44530/0 (socket says 172.21.15.5:44530) 2024-03-29T10:48:16.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febdbfff640 1 -- 172.21.15.5:0/3912995622 learned_addr learned my addr 172.21.15.5:0/3912995622 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febe1aed640 1 -- 172.21.15.5:0/3912995622 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc0aba70 con 0x7febdc0b0770 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febe1aed640 1 -- 172.21.15.5:0/3912995622 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc0aecb0 con 0x7febdc14a700 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febe1aed640 1 -- 172.21.15.5:0/3912995622 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc0af150 con 0x7febdc0ac5a0 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febe1aed640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febe1aed640 10 monclient(hunting): authenticate will time out at 909.966325s 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febda7fc640 1 -- 172.21.15.5:0/3912995622 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847398133 0 0) 0x7febdc0af150 con 0x7febdc0ac5a0 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febda7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febda7fc640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:48:16.708 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.725+0000 7febda7fc640 1 -- 172.21.15.5:0/3912995622 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc4003520 con 0x7febdc0ac5a0 2024-03-29T10:48:16.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.726+0000 7febda7fc640 1 -- 172.21.15.5:0/3912995622 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2906270524 0 0) 0x7febc4003520 con 0x7febdc0ac5a0 2024-03-29T10:48:16.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.726+0000 7febda7fc640 1 -- 172.21.15.5:0/3912995622 >> v1:172.21.15.177:6789/0 conn(0x7febdc14a700 legacy=0x7febdc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:16.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.726+0000 7febda7fc640 1 -- 172.21.15.5:0/3912995622 >> v1:172.21.15.5:6789/0 conn(0x7febdc0b0770 legacy=0x7febdc149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:16.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:16.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 1 monclient: found mon.noname-b 2024-03-29T10:48:16.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:16.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:48:16.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 1 -- 172.21.15.5:0/3912995622 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc0ac9d0 con 0x7febdc0ac5a0 2024-03-29T10:48:16.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 1 -- 172.21.15.5:0/3912995622 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7febd4001a10 con 0x7febdc0ac5a0 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febda7fc640 10 monclient: dump: 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:16.711 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:16.712 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:16.712 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febe1aed640 5 monclient: authenticate success, global_id 4250 2024-03-29T10:48:16.712 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.728+0000 7febe1aed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.729+0000 7febda7fc640 1 -- 172.21.15.5:0/3912995622 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febd4001be0 con 0x7febdc0ac5a0 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.729+0000 7febda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.729+0000 7febda7fc640 1 -- 172.21.15.5:0/3912995622 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7febd4002030 con 0x7febdc0ac5a0 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.729+0000 7febe1aed640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.729+0000 7febe1aed640 4 set_mon_vals no callback set 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.729+0000 7febda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.729+0000 7febda7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.729+0000 7febda7fc640 10 monclient: dump: 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:16.713 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.730+0000 7febe1aed640 10 monclient: shutdown 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.730+0000 7febe1aed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.730+0000 7febe1aed640 1 -- 172.21.15.5:0/3912995622 >> v1:172.21.15.5:6790/0 conn(0x7febdc0ac5a0 legacy=0x7febdc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.730+0000 7febe1aed640 1 -- 172.21.15.5:0/3912995622 shutdown_connections 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.730+0000 7febe1aed640 1 -- 172.21.15.5:0/3912995622 >> 172.21.15.5:0/3912995622 conn(0x7febdc06b660 msgr2=0x7febdc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.730+0000 7febe1aed640 1 -- 172.21.15.5:0/3912995622 shutdown_connections 2024-03-29T10:48:16.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.730+0000 7febe1aed640 1 -- 172.21.15.5:0/3912995622 wait complete. 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.731+0000 7febe1aed640 10 monclient: build_initial_monmap 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.731+0000 7febe1aed640 10 monclient: monmap: 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:48:16.715 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.731+0000 7febe1aed640 1 librados: starting msgr at 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.731+0000 7febe1aed640 1 librados: starting objecter 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.731+0000 7febe1aed640 1 Processor -- start 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.732+0000 7febe1aed640 1 -- start start 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.732+0000 7febe1aed640 1 librados: setting wanted keys 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.732+0000 7febe1aed640 1 librados: calling monclient init 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.732+0000 7febe1aed640 10 monclient: init 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febe1aed640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febe1aed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febe1aed640 10 monclient(hunting): picked mon.noname-b con 0x7febdc0ac5a0 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febe1aed640 10 monclient(hunting): picked mon.noname-a con 0x7febdc0b0770 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febe1aed640 10 monclient(hunting): picked mon.noname-c con 0x7febdc14a700 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febe1aed640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc1239d0 con 0x7febdc0b0770 2024-03-29T10:48:16.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febdbfff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7febdc14a700 0x7febdc1e6e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:50316/0 (socket says 172.21.15.5:50316) 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febdbfff640 1 -- 172.21.15.5:0/853422685 learned_addr learned my addr 172.21.15.5:0/853422685 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febc3fff640 10 client.?.objecter ms_handle_connect 0x7febdc14a700 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc1263f0 con 0x7febdc0ac5a0 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc1284a0 con 0x7febdc14a700 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febe1aed640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febe1aed640 10 monclient(hunting): authenticate will time out at 909.974796s 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febc3fff640 10 client.?.objecter ms_handle_connect 0x7febdc0ac5a0 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.733+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561262101 0 0) 0x7febdc1284a0 con 0x7febdc14a700 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:48:16.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc8004eb0 con 0x7febdc14a700 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524476471 0 0) 0x7febdc1263f0 con 0x7febdc0ac5a0 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc1284a0 con 0x7febdc0ac5a0 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 523075258 0 0) 0x7febc8004eb0 con 0x7febdc14a700 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febdc1263f0 con 0x7febdc14a700 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7febd4001ef0 con 0x7febdc14a700 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): dump: 2024-03-29T10:48:16.722 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4185125916 0 0) 0x7febdc1263f0 con 0x7febdc14a700 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 >> v1:172.21.15.177:6789/0 conn(0x7febdc0ac5a0 legacy=0x7febdc144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.734+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 >> v1:172.21.15.5:6789/0 conn(0x7febdc0b0770 legacy=0x7febdc126880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.735+0000 7febc3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:16.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febc3fff640 1 monclient: found mon.c 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febc3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc1dc1e0 con 0x7febdc14a700 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febc3fff640 10 monclient: _finish_auth 0 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febc3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febe1aed640 5 monclient: authenticate success, global_id 4253 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febe1aed640 10 monclient: _renew_subs 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febe1aed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febdc125e40 con 0x7febdc14a700 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febe1aed640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febe1aed640 10 monclient: _renew_subs 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febe1aed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febdc1458c0 con 0x7febdc14a700 2024-03-29T10:48:16.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.736+0000 7febe1aed640 1 librados: init done 2024-03-29T10:48:16.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.737+0000 7febe1aed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:16.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.737+0000 7febe1aed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:16.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.737+0000 7febe1aed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:16.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.737+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febdc1dc430 con 0x7febdc14a700 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.738+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febd4002090 con 0x7febdc14a700 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.738+0000 7febc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.738+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7febd4005110 con 0x7febdc14a700 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.738+0000 7febc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.738+0000 7febc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.738+0000 7febc3fff640 10 monclient: dump: 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:16.726 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.738+0000 7febd9ffb640 4 set_mon_vals no callback set 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.741+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7febd401b0e0 con 0x7febdc14a700 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.741+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7febd4084e10 con 0x7febdc14a700 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.741+0000 7febc3fff640 10 client.4253.objecter ms_dispatch 0x7febdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.741+0000 7febc3fff640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.741+0000 7febc3fff640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.741+0000 7febc3fff640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.742+0000 7febdb7fe640 10 monclient: get_auth_request con 0x7febc806dd30 auth_method 0 2024-03-29T10:48:16.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.742+0000 7febc3fff640 10 client.4253.objecter ms_handle_connect 0x7febc806dd30 2024-03-29T10:48:16.729 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.746+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febd40506f0 con 0x7febdc14a700 2024-03-29T10:48:16.729 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.746+0000 7febc3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:16.729 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:16.746+0000 7febc3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:17.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.064+0000 7febe1aed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:17.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.064+0000 7febe1aed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:17.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.064+0000 7febe1aed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:17.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.064+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febdc1265a0 con 0x7febdc14a700 2024-03-29T10:48:17.048 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.065+0000 7febc3fff640 1 -- 172.21.15.5:0/853422685 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7febdc1265a0 con 0x7febdc14a700 2024-03-29T10:48:17.048 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.065+0000 7febc3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:17.048 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.065+0000 7febc3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:17.048 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:48:17.048 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:17.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.068+0000 7febe1aed640 10 librados: watch_flush enter 2024-03-29T10:48:17.051 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.068+0000 7febe1aed640 10 librados: watch_flush exit 2024-03-29T10:48:17.051 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.068+0000 7febe1aed640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T10:48:17.051 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.068+0000 7febe1aed640 10 client.4253.objecter successfully canceled tick 2024-03-29T10:48:17.051 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.068+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 >> v1:172.21.15.5:6816/2847500151 conn(0x7febc806dd30 legacy=0x7febc80701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.051 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.068+0000 7febe1aed640 10 monclient: shutdown 2024-03-29T10:48:17.051 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.068+0000 7febe1aed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:17.051 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.068+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 >> v1:172.21.15.5:6790/0 conn(0x7febdc14a700 legacy=0x7febdc1e6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.051 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.069+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 shutdown_connections 2024-03-29T10:48:17.051 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.069+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 >> 172.21.15.5:0/853422685 conn(0x7febdc06b660 msgr2=0x7febdc0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:17.052 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.069+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 shutdown_connections 2024-03-29T10:48:17.052 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.069+0000 7febe1aed640 1 -- 172.21.15.5:0/853422685 wait complete. 2024-03-29T10:48:17.052 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:17.069+0000 7febe1aed640 1 librados: shutdown 2024-03-29T10:48:17.064 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:23.872 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:24.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.037+0000 7fe7cffff640 10 monclient: get_monmap_and_config 2024-03-29T10:48:24.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.037+0000 7fe7cffff640 10 monclient: build_initial_monmap 2024-03-29T10:48:24.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.037+0000 7fe7cffff640 10 monclient: monmap: 2024-03-29T10:48:24.020 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:24.020 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:24.020 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:48:24.038062+0000 2024-03-29T10:48:24.020 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:48:24.038062+0000 2024-03-29T10:48:24.020 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:24.020 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:24.020 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:24.020 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:48:24.020 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:48:24.020 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:24.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.038+0000 7fe7cffff640 1 Processor -- start 2024-03-29T10:48:24.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.038+0000 7fe7cffff640 1 -- start start 2024-03-29T10:48:24.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.038+0000 7fe7cffff640 10 monclient: init 2024-03-29T10:48:24.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.039+0000 7fe7cffff640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:24.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.039+0000 7fe7cffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:24.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.039+0000 7fe7cffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d00a9fe0 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:24.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.040+0000 7fe7cffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d00af600 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:24.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7ce7fc640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fe7d00af600 0x7fe7d00acc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:47841/0 (socket says 172.21.15.5:47841) 2024-03-29T10:48:24.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7ce7fc640 1 -- 172.21.15.5:0/3961532356 learned_addr learned my addr 172.21.15.5:0/3961532356 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:24.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d00ad340 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cffff640 1 -- 172.21.15.5:0/3961532356 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d00aaec0 con 0x7fe7d00ad340 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cffff640 1 -- 172.21.15.5:0/3961532356 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d00ab6a0 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cffff640 1 -- 172.21.15.5:0/3961532356 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d00b00a0 con 0x7fe7d00af600 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cffff640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cffff640 10 monclient(hunting): authenticate will time out at 917.283706s 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cdffb640 1 -- 172.21.15.5:0/3961532356 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499607692 0 0) 0x7fe7d00b00a0 con 0x7fe7d00af600 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cdffb640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.042+0000 7fe7cdffb640 1 -- 172.21.15.5:0/3961532356 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b8003520 con 0x7fe7d00af600 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 1 -- 172.21.15.5:0/3961532356 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336435591 0 0) 0x7fe7b8003520 con 0x7fe7d00af600 2024-03-29T10:48:24.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 1 -- 172.21.15.5:0/3961532356 >> v1:172.21.15.177:6789/0 conn(0x7fe7d00a9fe0 legacy=0x7fe7d013b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 1 -- 172.21.15.5:0/3961532356 >> v1:172.21.15.5:6789/0 conn(0x7fe7d00ad340 legacy=0x7fe7d0149c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 1 monclient: found mon.noname-b 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 1 -- 172.21.15.5:0/3961532356 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d013beb0 con 0x7fe7d00af600 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 1 -- 172.21.15.5:0/3961532356 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe7c0001690 con 0x7fe7d00af600 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: dump: 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:24.027 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:24.028 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:24.028 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:24.028 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:24.028 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:24.028 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:24.028 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:24.028 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:24.028 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:24.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cffff640 5 monclient: authenticate success, global_id 4256 2024-03-29T10:48:24.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:24.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 1 -- 172.21.15.5:0/3961532356 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7c0001860 con 0x7fe7d00af600 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 1 -- 172.21.15.5:0/3961532356 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe7c0001cb0 con 0x7fe7d00af600 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cdffb640 10 monclient: dump: 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:24.030 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:24.031 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:24.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cffff640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:24.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.043+0000 7fe7cffff640 4 set_mon_vals no callback set 2024-03-29T10:48:24.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.044+0000 7fe7cffff640 10 monclient: shutdown 2024-03-29T10:48:24.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.044+0000 7fe7cffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:24.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.044+0000 7fe7cffff640 1 -- 172.21.15.5:0/3961532356 >> v1:172.21.15.5:6790/0 conn(0x7fe7d00af600 legacy=0x7fe7d00acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:24.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.047+0000 7fe7cffff640 1 -- 172.21.15.5:0/3961532356 shutdown_connections 2024-03-29T10:48:24.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.047+0000 7fe7cffff640 1 -- 172.21.15.5:0/3961532356 >> 172.21.15.5:0/3961532356 conn(0x7fe7d00673e0 msgr2=0x7fe7d00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:24.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.047+0000 7fe7cffff640 1 -- 172.21.15.5:0/3961532356 shutdown_connections 2024-03-29T10:48:24.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.047+0000 7fe7cffff640 1 -- 172.21.15.5:0/3961532356 wait complete. 2024-03-29T10:48:24.033 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.048+0000 7fe7cffff640 10 monclient: build_initial_monmap 2024-03-29T10:48:24.033 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.048+0000 7fe7cffff640 10 monclient: monmap: 2024-03-29T10:48:24.033 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:24.033 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.049+0000 7fe7cffff640 1 librados: starting msgr at 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.049+0000 7fe7cffff640 1 librados: starting objecter 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.049+0000 7fe7cffff640 1 Processor -- start 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.049+0000 7fe7cffff640 1 -- start start 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.049+0000 7fe7cffff640 1 librados: setting wanted keys 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.049+0000 7fe7cffff640 1 librados: calling monclient init 2024-03-29T10:48:24.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.049+0000 7fe7cffff640 10 monclient: init 2024-03-29T10:48:24.035 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.050+0000 7fe7cffff640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:24.035 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.050+0000 7fe7cffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.051+0000 7fe7cffff640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d00a9fe0 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.052+0000 7fe7cffff640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d00ad340 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.052+0000 7fe7cffff640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d00af600 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.052+0000 7fe7cffff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d013ef60 con 0x7fe7d00af600 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.052+0000 7fe7cffff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d013ebf0 con 0x7fe7d00ad340 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.052+0000 7fe7cffff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d013f3d0 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.052+0000 7fe7cffff640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.052+0000 7fe7cffff640 10 monclient(hunting): authenticate will time out at 917.293170s 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.053+0000 7fe7ceffd640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fe7d00a9fe0 0x7fe7d01e7910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:47925/0 (socket says 172.21.15.5:47925) 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.053+0000 7fe7ceffd640 1 -- 172.21.15.5:0/325867590 learned_addr learned my addr 172.21.15.5:0/325867590 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.053+0000 7fe7b77fe640 10 client.?.objecter ms_handle_connect 0x7fe7d00a9fe0 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.054+0000 7fe7b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.054+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379274139 0 0) 0x7fe7d013f3d0 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.054+0000 7fe7b77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:24.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.054+0000 7fe7b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:24.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.054+0000 7fe7b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:24.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.054+0000 7fe7b77fe640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:48:24.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.054+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7bc004eb0 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3442521691 0 0) 0x7fe7bc004eb0 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7bc005300 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe7c4004040 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:24.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 10 monclient(hunting): dump: 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 10 client.?.objecter ms_handle_connect 0x7fe7d00af600 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:24.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2214082626 0 0) 0x7fe7bc005300 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 >> v1:172.21.15.177:6789/0 conn(0x7fe7d00ad340 legacy=0x7fe7d01e0d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 >> v1:172.21.15.5:6789/0 conn(0x7fe7d00af600 legacy=0x7fe7d01ef380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 1 monclient: found mon.c 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d006e230 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.055+0000 7fe7b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7cffff640 5 monclient: authenticate success, global_id 4259 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7cffff640 10 monclient: _renew_subs 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7cffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7cffff640 1 -- 172.21.15.5:0/325867590 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d006d8c0 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.044 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7cffff640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7cffff640 10 monclient: _renew_subs 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7cffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7cffff640 1 -- 172.21.15.5:0/325867590 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d01e1670 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7cffff640 1 librados: init done 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7c4000ed0 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe7c4004040 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7b77fe640 10 monclient: dump: 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:24.045 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.056+0000 7fe7cd7fa640 4 set_mon_vals no callback set 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.058+0000 7fe7b57fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.058+0000 7fe7b57fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.058+0000 7fe7b57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.058+0000 7fe7b57fa640 1 -- 172.21.15.5:0/325867590 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a8005080 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.058+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fe7c4001140 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.059+0000 7fe7ce7fc640 10 monclient: get_auth_request con 0x7fe7bc06e000 auth_method 0 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.059+0000 7fe7b77fe640 10 client.4259.objecter ms_handle_connect 0x7fe7bc06e000 2024-03-29T10:48:24.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.059+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fe7c4054600 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.059+0000 7fe7b77fe640 10 client.4259.objecter ms_dispatch 0x7fe7d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:24.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.059+0000 7fe7b77fe640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:24.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.059+0000 7fe7b77fe640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:24.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.059+0000 7fe7b77fe640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T10:48:24.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.068+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7c40508a0 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.068+0000 7fe7b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:24.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.068+0000 7fe7b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:24.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.396+0000 7fe7b57fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:24.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.396+0000 7fe7b57fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:24.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.396+0000 7fe7b57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:24.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.396+0000 7fe7b57fa640 1 -- 172.21.15.5:0/325867590 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7a8005330 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.398+0000 7fe7b77fe640 1 -- 172.21.15.5:0/325867590 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fe7c40508a0 con 0x7fe7d00a9fe0 2024-03-29T10:48:24.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.398+0000 7fe7b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:24.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.398+0000 7fe7b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:24.380 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:48:24.380 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:24.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.401+0000 7fe7b57fa640 10 librados: watch_flush enter 2024-03-29T10:48:24.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.403+0000 7fe7b57fa640 10 librados: watch_flush exit 2024-03-29T10:48:24.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.403+0000 7fe7b57fa640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T10:48:24.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.403+0000 7fe7b57fa640 10 client.4259.objecter successfully canceled tick 2024-03-29T10:48:24.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.403+0000 7fe7b57fa640 1 -- 172.21.15.5:0/325867590 >> v1:172.21.15.5:6816/2847500151 conn(0x7fe7bc06e000 legacy=0x7fe7bc0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:24.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.403+0000 7fe7b57fa640 10 monclient: shutdown 2024-03-29T10:48:24.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.403+0000 7fe7b57fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:24.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.403+0000 7fe7b57fa640 1 -- 172.21.15.5:0/325867590 >> v1:172.21.15.5:6790/0 conn(0x7fe7d00a9fe0 legacy=0x7fe7d01e7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:24.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.403+0000 7fe7b57fa640 1 -- 172.21.15.5:0/325867590 shutdown_connections 2024-03-29T10:48:24.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.403+0000 7fe7b57fa640 1 -- 172.21.15.5:0/325867590 >> 172.21.15.5:0/325867590 conn(0x7fe7d00673e0 msgr2=0x7fe7d0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:24.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.403+0000 7fe7b57fa640 1 -- 172.21.15.5:0/325867590 shutdown_connections 2024-03-29T10:48:24.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.404+0000 7fe7b57fa640 1 -- 172.21.15.5:0/325867590 wait complete. 2024-03-29T10:48:24.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:24.405+0000 7fe7b57fa640 1 librados: shutdown 2024-03-29T10:48:24.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:31.214 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:31.364 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.380+0000 7fa04a515640 10 monclient: get_monmap_and_config 2024-03-29T10:48:31.364 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.380+0000 7fa04a515640 10 monclient: build_initial_monmap 2024-03-29T10:48:31.364 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.380+0000 7fa04a515640 10 monclient: monmap: 2024-03-29T10:48:31.364 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:31.364 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:31.364 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:48:31.381789+0000 2024-03-29T10:48:31.364 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:48:31.381789+0000 2024-03-29T10:48:31.364 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:31.365 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:31.365 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:31.365 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:48:31.365 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:48:31.365 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:31.365 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.381+0000 7fa04a515640 1 Processor -- start 2024-03-29T10:48:31.365 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.381+0000 7fa04a515640 1 -- start start 2024-03-29T10:48:31.365 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.381+0000 7fa04a515640 10 monclient: init 2024-03-29T10:48:31.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.382+0000 7fa04a515640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:31.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.382+0000 7fa04a515640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:31.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.382+0000 7fa04a515640 10 monclient(hunting): picked mon.noname-b con 0x7fa0440ab360 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:31.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.382+0000 7fa04a515640 10 monclient(hunting): picked mon.noname-c con 0x7fa04414fb00 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:31.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.382+0000 7fa04a515640 10 monclient(hunting): picked mon.noname-a con 0x7fa044153090 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:31.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.382+0000 7fa04a515640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0440a94e0 con 0x7fa044153090 2024-03-29T10:48:31.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.382+0000 7fa04a515640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0440afb60 con 0x7fa04414fb00 2024-03-29T10:48:31.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.382+0000 7fa04a515640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0440aab80 con 0x7fa0440ab360 2024-03-29T10:48:31.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.382+0000 7fa04a515640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:31.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.382+0000 7fa04a515640 10 monclient(hunting): authenticate will time out at 924.623914s 2024-03-29T10:48:31.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.385+0000 7fa048ab2640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa044153090 0x7fa04415cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46363/0 (socket says 172.21.15.5:46363) 2024-03-29T10:48:31.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.385+0000 7fa048ab2640 1 -- 172.21.15.5:0/2592309968 learned_addr learned my addr 172.21.15.5:0/2592309968 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:31.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.388+0000 7fa042ffd640 1 -- 172.21.15.5:0/2592309968 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187591100 0 0) 0x7fa0440a94e0 con 0x7fa044153090 2024-03-29T10:48:31.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.388+0000 7fa042ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:31.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.388+0000 7fa042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:31.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.388+0000 7fa042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:31.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.388+0000 7fa042ffd640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:48:31.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.388+0000 7fa042ffd640 1 -- 172.21.15.5:0/2592309968 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa02c003520 con 0x7fa044153090 2024-03-29T10:48:31.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.389+0000 7fa042ffd640 1 -- 172.21.15.5:0/2592309968 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 592242874 0 0) 0x7fa02c003520 con 0x7fa044153090 2024-03-29T10:48:31.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.389+0000 7fa042ffd640 1 -- 172.21.15.5:0/2592309968 >> v1:172.21.15.5:6790/0 conn(0x7fa0440ab360 legacy=0x7fa0440ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.389+0000 7fa042ffd640 1 -- 172.21.15.5:0/2592309968 >> v1:172.21.15.177:6789/0 conn(0x7fa04414fb00 legacy=0x7fa044152980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.389+0000 7fa042ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.389+0000 7fa042ffd640 1 monclient: found mon.noname-a 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.389+0000 7fa042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.389+0000 7fa042ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.389+0000 7fa042ffd640 1 -- 172.21.15.5:0/2592309968 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0441568d0 con 0x7fa044153090 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa042ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa042ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa04a515640 5 monclient: authenticate success, global_id 4254 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa04a515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa042ffd640 1 -- 172.21.15.5:0/2592309968 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa03c001a10 con 0x7fa044153090 2024-03-29T10:48:31.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa042ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa042ffd640 10 monclient: dump: 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa04a515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa042ffd640 1 -- 172.21.15.5:0/2592309968 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa03c001ca0 con 0x7fa044153090 2024-03-29T10:48:31.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa042ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa04a515640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.390+0000 7fa04a515640 4 set_mon_vals no callback set 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.391+0000 7fa042ffd640 1 -- 172.21.15.5:0/2592309968 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa03c002030 con 0x7fa044153090 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.391+0000 7fa042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.391+0000 7fa042ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.391+0000 7fa042ffd640 10 monclient: dump: 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:31.377 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:31.378 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:31.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.391+0000 7fa04a515640 10 monclient: shutdown 2024-03-29T10:48:31.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.391+0000 7fa04a515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:31.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.391+0000 7fa04a515640 1 -- 172.21.15.5:0/2592309968 >> v1:172.21.15.5:6789/0 conn(0x7fa044153090 legacy=0x7fa04415cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:31.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.392+0000 7fa04a515640 1 -- 172.21.15.5:0/2592309968 shutdown_connections 2024-03-29T10:48:31.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.392+0000 7fa04a515640 1 -- 172.21.15.5:0/2592309968 >> 172.21.15.5:0/2592309968 conn(0x7fa0440b35e0 msgr2=0x7fa0440b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:31.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.392+0000 7fa04a515640 1 -- 172.21.15.5:0/2592309968 shutdown_connections 2024-03-29T10:48:31.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.393+0000 7fa04a515640 1 -- 172.21.15.5:0/2592309968 wait complete. 2024-03-29T10:48:31.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.393+0000 7fa04a515640 10 monclient: build_initial_monmap 2024-03-29T10:48:31.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.393+0000 7fa04a515640 10 monclient: monmap: 2024-03-29T10:48:31.380 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:31.380 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:31.380 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:48:31.380 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:48:31.380 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:31.380 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.394+0000 7fa04a515640 1 librados: starting msgr at 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.394+0000 7fa04a515640 1 librados: starting objecter 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.394+0000 7fa04a515640 1 Processor -- start 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.394+0000 7fa04a515640 1 -- start start 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.394+0000 7fa04a515640 1 librados: setting wanted keys 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.394+0000 7fa04a515640 1 librados: calling monclient init 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.394+0000 7fa04a515640 10 monclient: init 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.395+0000 7fa04a515640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.395+0000 7fa04a515640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.395+0000 7fa04a515640 10 monclient(hunting): picked mon.noname-c con 0x7fa0440ab360 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.395+0000 7fa04a515640 10 monclient(hunting): picked mon.noname-a con 0x7fa04414fb00 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:31.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.395+0000 7fa04a515640 10 monclient(hunting): picked mon.noname-b con 0x7fa044153090 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:31.382 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.395+0000 7fa04a515640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0441561d0 con 0x7fa04414fb00 2024-03-29T10:48:31.382 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.395+0000 7fa04a515640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0440700a0 con 0x7fa044153090 2024-03-29T10:48:31.382 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.395+0000 7fa04a515640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa044070380 con 0x7fa0440ab360 2024-03-29T10:48:31.382 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.395+0000 7fa04a515640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:31.382 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.395+0000 7fa04a515640 10 monclient(hunting): authenticate will time out at 924.636715s 2024-03-29T10:48:31.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.402+0000 7fa048ab2640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fa044153090 0x7fa0441ff8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:54319/0 (socket says 172.21.15.5:54319) 2024-03-29T10:48:31.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.402+0000 7fa048ab2640 1 -- 172.21.15.5:0/2766770115 learned_addr learned my addr 172.21.15.5:0/2766770115 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:31.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.402+0000 7fa040ff9640 10 client.?.objecter ms_handle_connect 0x7fa044153090 2024-03-29T10:48:31.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:31.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 10 client.?.objecter ms_handle_connect 0x7fa0440ab360 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251110915 0 0) 0x7fa044070380 con 0x7fa0440ab360 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa030004eb0 con 0x7fa0440ab360 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 10 client.?.objecter ms_handle_connect 0x7fa04414fb00 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1346059152 0 0) 0x7fa030004eb0 con 0x7fa0440ab360 2024-03-29T10:48:31.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa044070380 con 0x7fa0440ab360 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.403+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa034001870 con 0x7fa0440ab360 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 10 monclient(hunting): dump: 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:31.390 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285401720 0 0) 0x7fa0441561d0 con 0x7fa04414fb00 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa030004eb0 con 0x7fa04414fb00 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3021112634 0 0) 0x7fa044070380 con 0x7fa0440ab360 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 >> v1:172.21.15.177:6789/0 conn(0x7fa044153090 legacy=0x7fa0441ff8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 >> v1:172.21.15.5:6789/0 conn(0x7fa04414fb00 legacy=0x7fa044070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.404+0000 7fa040ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:48:31.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 1 monclient: found mon.c 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa044070660 con 0x7fa0440ab360 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 10 monclient: _finish_auth 0 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa04a515640 5 monclient: authenticate success, global_id 4262 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa04a515640 10 monclient: _renew_subs 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa04a515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa04a515640 1 -- 172.21.15.5:0/2766770115 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04408aad0 con 0x7fa0440ab360 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa04a515640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa04a515640 10 monclient: _renew_subs 2024-03-29T10:48:31.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa04a515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:31.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa04a515640 1 -- 172.21.15.5:0/2766770115 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa044072560 con 0x7fa0440ab360 2024-03-29T10:48:31.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa04a515640 1 librados: init done 2024-03-29T10:48:31.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0340035f0 con 0x7fa0440ab360 2024-03-29T10:48:31.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa034003900 con 0x7fa0440ab360 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa040ff9640 10 monclient: dump: 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:31.395 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.405+0000 7fa041ffb640 4 set_mon_vals no callback set 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.408+0000 7fa04a515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.408+0000 7fa04a515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.408+0000 7fa04a515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.408+0000 7fa04a515640 1 -- 172.21.15.5:0/2766770115 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa024005080 con 0x7fa0440ab360 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.409+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fa0340198d0 con 0x7fa0440ab360 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.411+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fa034083780 con 0x7fa0440ab360 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.411+0000 7fa0437fe640 10 monclient: get_auth_request con 0x7fa03006de10 auth_method 0 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.411+0000 7fa040ff9640 10 client.4262.objecter ms_dispatch 0x7fa044000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.411+0000 7fa040ff9640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.411+0000 7fa040ff9640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.411+0000 7fa040ff9640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T10:48:31.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.411+0000 7fa040ff9640 10 client.4262.objecter ms_handle_connect 0x7fa03006de10 2024-03-29T10:48:31.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.421+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa03404f060 con 0x7fa0440ab360 2024-03-29T10:48:31.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.421+0000 7fa040ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:31.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.421+0000 7fa040ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:31.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.747+0000 7fa04a515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:31.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.747+0000 7fa04a515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:31.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.747+0000 7fa04a515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:31.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.747+0000 7fa04a515640 1 -- 172.21.15.5:0/2766770115 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa024005330 con 0x7fa0440ab360 2024-03-29T10:48:31.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.751+0000 7fa040ff9640 1 -- 172.21.15.5:0/2766770115 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fa034050f90 con 0x7fa0440ab360 2024-03-29T10:48:31.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.752+0000 7fa040ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:31.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.752+0000 7fa040ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:31.734 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:48:31.734 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:31.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.756+0000 7fa04a515640 10 librados: watch_flush enter 2024-03-29T10:48:31.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.756+0000 7fa04a515640 10 librados: watch_flush exit 2024-03-29T10:48:31.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.756+0000 7fa04a515640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T10:48:31.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.756+0000 7fa04a515640 10 client.4262.objecter successfully canceled tick 2024-03-29T10:48:31.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.757+0000 7fa04a515640 1 -- 172.21.15.5:0/2766770115 >> v1:172.21.15.5:6816/2847500151 conn(0x7fa03006de10 legacy=0x7fa0300702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:31.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.757+0000 7fa04a515640 10 monclient: shutdown 2024-03-29T10:48:31.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.757+0000 7fa04a515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:31.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.757+0000 7fa04a515640 1 -- 172.21.15.5:0/2766770115 >> v1:172.21.15.5:6790/0 conn(0x7fa0440ab360 legacy=0x7fa04408a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:31.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.758+0000 7fa04a515640 1 -- 172.21.15.5:0/2766770115 shutdown_connections 2024-03-29T10:48:31.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.758+0000 7fa04a515640 1 -- 172.21.15.5:0/2766770115 >> 172.21.15.5:0/2766770115 conn(0x7fa0440b35e0 msgr2=0x7fa0440a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:31.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.758+0000 7fa04a515640 1 -- 172.21.15.5:0/2766770115 shutdown_connections 2024-03-29T10:48:31.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.759+0000 7fa04a515640 1 -- 172.21.15.5:0/2766770115 wait complete. 2024-03-29T10:48:31.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:31.759+0000 7fa04a515640 1 librados: shutdown 2024-03-29T10:48:31.761 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:38.568 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:38.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.728+0000 7f5c60d66640 10 monclient: get_monmap_and_config 2024-03-29T10:48:38.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.728+0000 7f5c60d66640 10 monclient: build_initial_monmap 2024-03-29T10:48:38.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.728+0000 7f5c60d66640 10 monclient: monmap: 2024-03-29T10:48:38.711 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:38.711 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:38.711 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:48:38.729467+0000 2024-03-29T10:48:38.711 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:48:38.729467+0000 2024-03-29T10:48:38.711 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:38.711 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:38.711 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:38.712 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:48:38.712 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:48:38.712 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:38.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.732+0000 7f5c60d66640 1 Processor -- start 2024-03-29T10:48:38.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.733+0000 7f5c60d66640 1 -- start start 2024-03-29T10:48:38.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.733+0000 7f5c60d66640 10 monclient: init 2024-03-29T10:48:38.717 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.734+0000 7f5c60d66640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:38.717 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.734+0000 7f5c60d66640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:38.717 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.734+0000 7f5c60d66640 10 monclient(hunting): picked mon.noname-b con 0x7f5c54006a70 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:38.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.737+0000 7f5c60d66640 10 monclient(hunting): picked mon.noname-c con 0x7f5c5402b100 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:38.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.737+0000 7f5c60d66640 10 monclient(hunting): picked mon.noname-a con 0x7f5c54029c30 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:38.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.737+0000 7f5c60d66640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5402a270 con 0x7f5c54029c30 2024-03-29T10:48:38.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.737+0000 7f5c60d66640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5402c440 con 0x7f5c5402b100 2024-03-29T10:48:38.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.737+0000 7f5c60d66640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540c9440 con 0x7f5c54006a70 2024-03-29T10:48:38.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.737+0000 7f5c60d66640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:38.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.737+0000 7f5c60d66640 10 monclient(hunting): authenticate will time out at 931.978970s 2024-03-29T10:48:38.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.738+0000 7f5c5affd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f5c5402b100 0x7f5c540cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:43641/0 (socket says 172.21.15.5:43641) 2024-03-29T10:48:38.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.738+0000 7f5c5affd640 1 -- 172.21.15.5:0/3726814016 learned_addr learned my addr 172.21.15.5:0/3726814016 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:38.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 1 -- 172.21.15.5:0/3726814016 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305586652 0 0) 0x7f5c5402c440 con 0x7f5c5402b100 2024-03-29T10:48:38.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:38.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:38.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:38.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:48:38.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 1 -- 172.21.15.5:0/3726814016 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c4c003520 con 0x7f5c5402b100 2024-03-29T10:48:38.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 1 -- 172.21.15.5:0/3726814016 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520576566 0 0) 0x7f5c4c003520 con 0x7f5c5402b100 2024-03-29T10:48:38.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 1 -- 172.21.15.5:0/3726814016 >> v1:172.21.15.5:6790/0 conn(0x7f5c54006a70 legacy=0x7f5c540c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:38.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 1 -- 172.21.15.5:0/3726814016 >> v1:172.21.15.5:6789/0 conn(0x7f5c54029c30 legacy=0x7f5c540d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:38.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:38.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:48:38.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:38.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:48:38.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 1 -- 172.21.15.5:0/3726814016 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c540c95d0 con 0x7f5c5402b100 2024-03-29T10:48:38.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:38.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:38.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:38.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 1 -- 172.21.15.5:0/3726814016 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5c5c04f730 con 0x7f5c5402b100 2024-03-29T10:48:38.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:38.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.741+0000 7f5c5a7fc640 10 monclient: dump: 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:38.726 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:38.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c60d66640 5 monclient: authenticate success, global_id 4246 2024-03-29T10:48:38.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c60d66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:38.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c5a7fc640 1 -- 172.21.15.5:0/3726814016 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c5c075f10 con 0x7f5c5402b100 2024-03-29T10:48:38.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:38.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c5a7fc640 1 -- 172.21.15.5:0/3726814016 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5c5c0758b0 con 0x7f5c5402b100 2024-03-29T10:48:38.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:38.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:38.729 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c5a7fc640 10 monclient: dump: 2024-03-29T10:48:38.729 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:38.729 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:38.729 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:38.729 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:38.729 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:38.729 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:38.729 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:38.730 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:38.730 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:38.730 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:38.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c60d66640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:38.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c60d66640 4 set_mon_vals no callback set 2024-03-29T10:48:38.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c60d66640 10 monclient: shutdown 2024-03-29T10:48:38.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c60d66640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:38.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.743+0000 7f5c60d66640 1 -- 172.21.15.5:0/3726814016 >> v1:172.21.15.177:6789/0 conn(0x7f5c5402b100 legacy=0x7f5c540cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:38.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.744+0000 7f5c60d66640 1 -- 172.21.15.5:0/3726814016 shutdown_connections 2024-03-29T10:48:38.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.744+0000 7f5c60d66640 1 -- 172.21.15.5:0/3726814016 >> 172.21.15.5:0/3726814016 conn(0x7f5c540058f0 msgr2=0x7f5c54005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:38.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.747+0000 7f5c60d66640 1 -- 172.21.15.5:0/3726814016 shutdown_connections 2024-03-29T10:48:38.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.747+0000 7f5c60d66640 1 -- 172.21.15.5:0/3726814016 wait complete. 2024-03-29T10:48:38.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.748+0000 7f5c60d66640 10 monclient: build_initial_monmap 2024-03-29T10:48:38.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.748+0000 7f5c60d66640 10 monclient: monmap: 2024-03-29T10:48:38.732 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:38.732 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:38.732 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:48:38.732 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:48:38.732 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:38.732 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:38.732 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:38.733 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:48:38.733 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:48:38.733 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:38.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.748+0000 7f5c60d66640 1 librados: starting msgr at 2024-03-29T10:48:38.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.748+0000 7f5c60d66640 1 librados: starting objecter 2024-03-29T10:48:38.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.748+0000 7f5c60d66640 1 Processor -- start 2024-03-29T10:48:38.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.749+0000 7f5c60d66640 1 -- start start 2024-03-29T10:48:38.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.749+0000 7f5c60d66640 1 librados: setting wanted keys 2024-03-29T10:48:38.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.749+0000 7f5c60d66640 1 librados: calling monclient init 2024-03-29T10:48:38.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.749+0000 7f5c60d66640 10 monclient: init 2024-03-29T10:48:38.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.750+0000 7f5c60d66640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:38.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.750+0000 7f5c60d66640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:38.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.750+0000 7f5c60d66640 10 monclient(hunting): picked mon.noname-c con 0x7f5c54006a70 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:38.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.753+0000 7f5c60d66640 10 monclient(hunting): picked mon.noname-b con 0x7f5c54029c30 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:38.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c5b7fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f5c54006a70 0x7f5c540ccee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:41849/0 (socket says 172.21.15.5:41849) 2024-03-29T10:48:38.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c60d66640 10 monclient(hunting): picked mon.noname-a con 0x7f5c540ce220 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:38.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c60d66640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540cbae0 con 0x7f5c540ce220 2024-03-29T10:48:38.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c60d66640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540cbdc0 con 0x7f5c54029c30 2024-03-29T10:48:38.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c60d66640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c54169ac0 con 0x7f5c54006a70 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c60d66640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c60d66640 10 monclient(hunting): authenticate will time out at 931.996225s 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c5b7fe640 1 -- 172.21.15.5:0/839953714 learned_addr learned my addr 172.21.15.5:0/839953714 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c47fff640 10 client.?.objecter ms_handle_connect 0x7f5c54006a70 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c47fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896728661 0 0) 0x7f5c54169ac0 con 0x7f5c54006a70 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c47fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c47fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c47fff640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:48:38.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.755+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c40004eb0 con 0x7f5c54006a70 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.756+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1812379876 0 0) 0x7f5c40004eb0 con 0x7f5c54006a70 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.756+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c54169ac0 con 0x7f5c54006a70 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.756+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5c48001990 con 0x7f5c54006a70 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.756+0000 7f5c47fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.756+0000 7f5c47fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.756+0000 7f5c47fff640 10 monclient(hunting): dump: 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:38.743 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.756+0000 7f5c47fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.757+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1405430622 0 0) 0x7f5c54169ac0 con 0x7f5c54006a70 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.757+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 >> v1:172.21.15.177:6789/0 conn(0x7f5c54029c30 legacy=0x7f5c540cdb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 >> v1:172.21.15.5:6789/0 conn(0x7f5c540ce220 legacy=0x7f5c54174be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 1 monclient: found mon.c 2024-03-29T10:48:38.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c540cd5f0 con 0x7f5c54006a70 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 10 monclient: _finish_auth 0 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 10 client.?.objecter ms_handle_connect 0x7f5c540ce220 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c47fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c60d66640 5 monclient: authenticate success, global_id 4268 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c60d66640 10 monclient: _renew_subs 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c60d66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:38.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c60d66640 1 -- 172.21.15.5:0/839953714 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c54169fe0 con 0x7f5c54006a70 2024-03-29T10:48:38.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c60d66640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:38.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c60d66640 10 monclient: _renew_subs 2024-03-29T10:48:38.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c60d66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:38.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c60d66640 1 -- 172.21.15.5:0/839953714 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c540cca10 con 0x7f5c54006a70 2024-03-29T10:48:38.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.758+0000 7f5c60d66640 1 librados: init done 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.759+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c48001c90 con 0x7f5c54006a70 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.759+0000 7f5c47fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.759+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5c48001990 con 0x7f5c54006a70 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.759+0000 7f5c47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.759+0000 7f5c47fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.759+0000 7f5c47fff640 10 monclient: dump: 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:38.747 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:38.748 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:38.748 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:38.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.760+0000 7f5c59ffb640 4 set_mon_vals no callback set 2024-03-29T10:48:38.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.760+0000 7f5c60d66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:38.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.760+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f5c48003940 con 0x7f5c54006a70 2024-03-29T10:48:38.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.761+0000 7f5c60d66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:38.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.761+0000 7f5c60d66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:38.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.761+0000 7f5c60d66640 1 -- 172.21.15.5:0/839953714 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c005080 con 0x7f5c54006a70 2024-03-29T10:48:38.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.761+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f5c48054510 con 0x7f5c54006a70 2024-03-29T10:48:38.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.761+0000 7f5c47fff640 10 client.4268.objecter ms_dispatch 0x7f5c5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:38.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.761+0000 7f5c47fff640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:38.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.761+0000 7f5c47fff640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:38.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.761+0000 7f5c47fff640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-29T10:48:38.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.761+0000 7f5c5affd640 10 monclient: get_auth_request con 0x7f5c4006dfc0 auth_method 0 2024-03-29T10:48:38.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.762+0000 7f5c47fff640 10 client.4268.objecter ms_handle_connect 0x7f5c4006dfc0 2024-03-29T10:48:38.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.770+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c480507b0 con 0x7f5c54006a70 2024-03-29T10:48:38.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.770+0000 7f5c47fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:38.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:38.770+0000 7f5c47fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:39.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.098+0000 7f5c60d66640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:39.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.098+0000 7f5c60d66640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:39.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.098+0000 7f5c60d66640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:39.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.098+0000 7f5c60d66640 1 -- 172.21.15.5:0/839953714 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c3c005330 con 0x7f5c54006a70 2024-03-29T10:48:39.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.099+0000 7f5c47fff640 1 -- 172.21.15.5:0/839953714 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f5c480507b0 con 0x7f5c54006a70 2024-03-29T10:48:39.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.099+0000 7f5c47fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:39.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.099+0000 7f5c47fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:39.082 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:48:39.082 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:39.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.103+0000 7f5c60d66640 10 librados: watch_flush enter 2024-03-29T10:48:39.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.103+0000 7f5c60d66640 10 librados: watch_flush exit 2024-03-29T10:48:39.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.103+0000 7f5c60d66640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-29T10:48:39.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.103+0000 7f5c60d66640 10 client.4268.objecter successfully canceled tick 2024-03-29T10:48:39.088 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.106+0000 7f5c60d66640 1 -- 172.21.15.5:0/839953714 >> v1:172.21.15.5:6816/2847500151 conn(0x7f5c4006dfc0 legacy=0x7f5c40070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.088 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.106+0000 7f5c60d66640 10 monclient: shutdown 2024-03-29T10:48:39.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.107+0000 7f5c60d66640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:39.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.107+0000 7f5c60d66640 1 -- 172.21.15.5:0/839953714 >> v1:172.21.15.5:6790/0 conn(0x7f5c54006a70 legacy=0x7f5c540ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.090 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.108+0000 7f5c60d66640 1 -- 172.21.15.5:0/839953714 shutdown_connections 2024-03-29T10:48:39.090 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.108+0000 7f5c60d66640 1 -- 172.21.15.5:0/839953714 >> 172.21.15.5:0/839953714 conn(0x7f5c540058f0 msgr2=0x7f5c5415fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:39.090 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.108+0000 7f5c60d66640 1 -- 172.21.15.5:0/839953714 shutdown_connections 2024-03-29T10:48:39.090 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.108+0000 7f5c60d66640 1 -- 172.21.15.5:0/839953714 wait complete. 2024-03-29T10:48:39.090 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:39.109+0000 7f5c60d66640 1 librados: shutdown 2024-03-29T10:48:39.111 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:45.918 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:46.065 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.083+0000 7fda2822e640 10 monclient: get_monmap_and_config 2024-03-29T10:48:46.065 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.083+0000 7fda2822e640 10 monclient: build_initial_monmap 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.083+0000 7fda2822e640 10 monclient: monmap: 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:48:46.084466+0000 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:48:46.084466+0000 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:48:46.066 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:46.068 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.086+0000 7fda2822e640 1 Processor -- start 2024-03-29T10:48:46.069 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.086+0000 7fda2822e640 1 -- start start 2024-03-29T10:48:46.069 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.086+0000 7fda2822e640 10 monclient: init 2024-03-29T10:48:46.069 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.087+0000 7fda2822e640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:46.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.087+0000 7fda2822e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:46.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.087+0000 7fda2822e640 10 monclient(hunting): picked mon.noname-b con 0x7fda200b0770 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:46.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.087+0000 7fda2822e640 10 monclient(hunting): picked mon.noname-a con 0x7fda2014a700 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:46.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.087+0000 7fda2822e640 10 monclient(hunting): picked mon.noname-c con 0x7fda200ac5a0 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:46.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.087+0000 7fda2822e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200aba70 con 0x7fda2014a700 2024-03-29T10:48:46.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.087+0000 7fda2822e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200aecb0 con 0x7fda200ac5a0 2024-03-29T10:48:46.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.087+0000 7fda2822e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200af150 con 0x7fda200b0770 2024-03-29T10:48:46.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.087+0000 7fda2822e640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:46.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.087+0000 7fda2822e640 10 monclient(hunting): authenticate will time out at 939.329049s 2024-03-29T10:48:46.077 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.095+0000 7fda267cb640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fda200ac5a0 0x7fda20151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:46879/0 (socket says 172.21.15.5:46879) 2024-03-29T10:48:46.077 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.095+0000 7fda257c9640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fda2014a700 0x7fda200abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50773/0 (socket says 172.21.15.5:50773) 2024-03-29T10:48:46.077 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.095+0000 7fda267cb640 1 -- 172.21.15.5:0/3435349383 learned_addr learned my addr 172.21.15.5:0/3435349383 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:46.077 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.095+0000 7fda24fc8640 1 -- 172.21.15.5:0/3435349383 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229897147 0 0) 0x7fda200aba70 con 0x7fda2014a700 2024-03-29T10:48:46.077 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.095+0000 7fda24fc8640 10 monclient(hunting): _negotiate 2024-03-29T10:48:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.095+0000 7fda24fc8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.095+0000 7fda24fc8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.095+0000 7fda24fc8640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:48:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.095+0000 7fda24fc8640 1 -- 172.21.15.5:0/3435349383 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c003520 con 0x7fda2014a700 2024-03-29T10:48:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.096+0000 7fda24fc8640 1 -- 172.21.15.5:0/3435349383 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3958451454 0 0) 0x7fda0c003520 con 0x7fda2014a700 2024-03-29T10:48:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.096+0000 7fda24fc8640 1 -- 172.21.15.5:0/3435349383 >> v1:172.21.15.5:6790/0 conn(0x7fda200b0770 legacy=0x7fda20149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.098+0000 7fda267cb640 1 -- 172.21.15.5:0/3435349383 >> v1:172.21.15.177:6789/0 conn(0x7fda200ac5a0 legacy=0x7fda20151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T10:48:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.098+0000 7fda267cb640 1 -- 172.21.15.5:0/3435349383 >> v1:172.21.15.177:6789/0 conn(0x7fda200ac5a0 legacy=0x7fda20151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:48:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda267cb640 1 --1- 172.21.15.5:0/3435349383 >> v1:172.21.15.177:6789/0 conn(0x7fda200ac5a0 0x7fda20151f00 :-1 s=OPENED pgs=111903 cs=1 l=1).handle_message read tag failed 2024-03-29T10:48:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda267cb640 1 --1- 172.21.15.5:0/3435349383 >> v1:172.21.15.177:6789/0 conn(0x7fda200ac5a0 0x7fda20151f00 :-1 s=OPENED pgs=111903 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 1 -- 172.21.15.5:0/3435349383 >> v1:172.21.15.177:6789/0 conn(0x7fda200ac5a0 legacy=0x7fda20151f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 10 monclient: _finish_hunting 0 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 1 monclient: found mon.noname-a 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 1 -- 172.21.15.5:0/3435349383 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200ac9d0 con 0x7fda2014a700 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 10 monclient: _finish_auth 0 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 10 monclient: _check_auth_tickets 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda2822e640 5 monclient: authenticate success, global_id 4260 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda2822e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 10 monclient: ms_handle_reset stray mon v1:172.21.15.177:6789/0 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 1 -- 172.21.15.5:0/3435349383 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fda10001690 con 0x7fda2014a700 2024-03-29T10:48:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 10 monclient: dump: 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda2822e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:46.086 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 1 -- 172.21.15.5:0/3435349383 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda10001920 con 0x7fda2014a700 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda24fc8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda2822e640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.099+0000 7fda2822e640 4 set_mon_vals no callback set 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.100+0000 7fda2822e640 10 monclient: shutdown 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.100+0000 7fda2822e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.100+0000 7fda24fc8640 1 -- 172.21.15.5:0/3435349383 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fda10001cb0 con 0x7fda2014a700 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.100+0000 7fda2822e640 1 -- 172.21.15.5:0/3435349383 >> v1:172.21.15.5:6789/0 conn(0x7fda2014a700 legacy=0x7fda200abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.100+0000 7fda24fc8640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.101+0000 7fda2822e640 1 -- 172.21.15.5:0/3435349383 shutdown_connections 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.101+0000 7fda2822e640 1 -- 172.21.15.5:0/3435349383 >> 172.21.15.5:0/3435349383 conn(0x7fda2006b660 msgr2=0x7fda2006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.102+0000 7fda2822e640 1 -- 172.21.15.5:0/3435349383 shutdown_connections 2024-03-29T10:48:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.102+0000 7fda2822e640 1 -- 172.21.15.5:0/3435349383 wait complete. 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.103+0000 7fda2822e640 10 monclient: build_initial_monmap 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.103+0000 7fda2822e640 10 monclient: monmap: 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:48:46.088 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:48:46.089 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:46.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.103+0000 7fda2822e640 1 librados: starting msgr at 2024-03-29T10:48:46.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.103+0000 7fda2822e640 1 librados: starting objecter 2024-03-29T10:48:46.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.103+0000 7fda2822e640 1 Processor -- start 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.103+0000 7fda2822e640 1 -- start start 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.103+0000 7fda2822e640 1 librados: setting wanted keys 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.103+0000 7fda2822e640 1 librados: calling monclient init 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.103+0000 7fda2822e640 10 monclient: init 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.104+0000 7fda2822e640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.104+0000 7fda2822e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.104+0000 7fda2822e640 10 monclient(hunting): picked mon.noname-a con 0x7fda200ac5a0 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.104+0000 7fda2822e640 10 monclient(hunting): picked mon.noname-c con 0x7fda200b0770 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.104+0000 7fda2822e640 10 monclient(hunting): picked mon.noname-b con 0x7fda2014a700 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.104+0000 7fda2822e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2012a350 con 0x7fda200ac5a0 2024-03-29T10:48:46.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.104+0000 7fda2822e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda20127fd0 con 0x7fda2014a700 2024-03-29T10:48:46.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.104+0000 7fda2822e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda20127c60 con 0x7fda200b0770 2024-03-29T10:48:46.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.104+0000 7fda2822e640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:46.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.104+0000 7fda2822e640 10 monclient(hunting): authenticate will time out at 939.345688s 2024-03-29T10:48:46.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.110+0000 7fda25fca640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fda200ac5a0 0x7fda20146d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50875/0 (socket says 172.21.15.5:50875) 2024-03-29T10:48:46.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.110+0000 7fda25fca640 1 -- 172.21.15.5:0/560440782 learned_addr learned my addr 172.21.15.5:0/560440782 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.110+0000 7fda167fc640 10 client.?.objecter ms_handle_connect 0x7fda200b0770 2024-03-29T10:48:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.110+0000 7fda167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 client.?.objecter ms_handle_connect 0x7fda2014a700 2024-03-29T10:48:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 client.?.objecter ms_handle_connect 0x7fda200ac5a0 2024-03-29T10:48:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362768534 0 0) 0x7fda20127c60 con 0x7fda200b0770 2024-03-29T10:48:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda04004eb0 con 0x7fda200b0770 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997328543 0 0) 0x7fda20127fd0 con 0x7fda2014a700 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda20127c60 con 0x7fda2014a700 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823396014 0 0) 0x7fda2012a350 con 0x7fda200ac5a0 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda20127fd0 con 0x7fda200ac5a0 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.111+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3869497838 0 0) 0x7fda04004eb0 con 0x7fda200b0770 2024-03-29T10:48:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda2012a350 con 0x7fda200b0770 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fda10001610 con 0x7fda200b0770 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 10 monclient(hunting): dump: 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1605359149 0 0) 0x7fda2012a350 con 0x7fda200b0770 2024-03-29T10:48:46.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 >> v1:172.21.15.177:6789/0 conn(0x7fda2014a700 legacy=0x7fda20128c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:46.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 >> v1:172.21.15.5:6789/0 conn(0x7fda200ac5a0 legacy=0x7fda20146d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:46.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:46.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 1 monclient: found mon.c 2024-03-29T10:48:46.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:46.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:46.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20147a70 con 0x7fda200b0770 2024-03-29T10:48:46.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:46.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:46.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda2822e640 5 monclient: authenticate success, global_id 4271 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda2822e640 10 monclient: _renew_subs 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda2822e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda2822e640 1 -- 172.21.15.5:0/560440782 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda201df590 con 0x7fda200b0770 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda2822e640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda2822e640 10 monclient: _renew_subs 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda2822e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda2822e640 1 -- 172.21.15.5:0/560440782 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda201ded50 con 0x7fda200b0770 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.112+0000 7fda2822e640 1 librados: init done 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.115+0000 7fda2822e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:46.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.115+0000 7fda2822e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:46.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.115+0000 7fda2822e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:46.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.115+0000 7fda2822e640 1 -- 172.21.15.5:0/560440782 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda20147c60 con 0x7fda200b0770 2024-03-29T10:48:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.119+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda10004a90 con 0x7fda200b0770 2024-03-29T10:48:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.119+0000 7fda167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.119+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fda10004c00 con 0x7fda200b0770 2024-03-29T10:48:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.119+0000 7fda167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.119+0000 7fda167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.119+0000 7fda167fc640 10 monclient: dump: 2024-03-29T10:48:46.102 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.119+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fda1001abd0 con 0x7fda200b0770 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.119+0000 7fda17fff640 4 set_mon_vals no callback set 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.120+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fda10084920 con 0x7fda200b0770 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.120+0000 7fda167fc640 10 client.4271.objecter ms_dispatch 0x7fda20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.120+0000 7fda167fc640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.120+0000 7fda167fc640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.120+0000 7fda167fc640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-29T10:48:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.120+0000 7fda25fca640 10 monclient: get_auth_request con 0x7fda0406def0 auth_method 0 2024-03-29T10:48:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.121+0000 7fda167fc640 10 client.4271.objecter ms_handle_connect 0x7fda0406def0 2024-03-29T10:48:46.107 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.125+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda10050280 con 0x7fda200b0770 2024-03-29T10:48:46.107 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.125+0000 7fda167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:46.107 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.125+0000 7fda167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:46.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.464+0000 7fda2822e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:46.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.464+0000 7fda2822e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:46.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.464+0000 7fda2822e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:48:46.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.464+0000 7fda2822e640 1 -- 172.21.15.5:0/560440782 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda20128160 con 0x7fda200b0770 2024-03-29T10:48:46.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.465+0000 7fda167fc640 1 -- 172.21.15.5:0/560440782 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fda100521b0 con 0x7fda200b0770 2024-03-29T10:48:46.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.465+0000 7fda167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:46.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.465+0000 7fda167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:46.448 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:48:46.448 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:46.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.470+0000 7fda2822e640 10 librados: watch_flush enter 2024-03-29T10:48:46.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.470+0000 7fda2822e640 10 librados: watch_flush exit 2024-03-29T10:48:46.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.470+0000 7fda2822e640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-29T10:48:46.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.470+0000 7fda2822e640 10 client.4271.objecter successfully canceled tick 2024-03-29T10:48:46.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.470+0000 7fda2822e640 1 -- 172.21.15.5:0/560440782 >> v1:172.21.15.5:6816/2847500151 conn(0x7fda0406def0 legacy=0x7fda040703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:46.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.470+0000 7fda2822e640 10 monclient: shutdown 2024-03-29T10:48:46.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.470+0000 7fda2822e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:46.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.470+0000 7fda2822e640 1 -- 172.21.15.5:0/560440782 >> v1:172.21.15.5:6790/0 conn(0x7fda200b0770 legacy=0x7fda201284f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:46.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.471+0000 7fda2822e640 1 -- 172.21.15.5:0/560440782 shutdown_connections 2024-03-29T10:48:46.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.471+0000 7fda2822e640 1 -- 172.21.15.5:0/560440782 >> 172.21.15.5:0/560440782 conn(0x7fda2006b660 msgr2=0x7fda200abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:46.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.471+0000 7fda2822e640 1 -- 172.21.15.5:0/560440782 shutdown_connections 2024-03-29T10:48:46.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.471+0000 7fda2822e640 1 -- 172.21.15.5:0/560440782 wait complete. 2024-03-29T10:48:46.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:46.471+0000 7fda2822e640 1 librados: shutdown 2024-03-29T10:48:46.475 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:53.282 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.455+0000 7fef73fff640 10 monclient: get_monmap_and_config 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.455+0000 7fef73fff640 10 monclient: build_initial_monmap 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.455+0000 7fef73fff640 10 monclient: monmap: 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:48:53.456530+0000 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:48:53.456530+0000 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:53.438 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:48:53.439 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:48:53.439 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:53.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.456+0000 7fef73fff640 1 Processor -- start 2024-03-29T10:48:53.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.456+0000 7fef73fff640 1 -- start start 2024-03-29T10:48:53.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.456+0000 7fef73fff640 10 monclient: init 2024-03-29T10:48:53.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.457+0000 7fef73fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:53.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.457+0000 7fef73fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:53.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.457+0000 7fef73fff640 10 monclient(hunting): picked mon.noname-a con 0x7fef74071f40 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:53.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.457+0000 7fef73fff640 10 monclient(hunting): picked mon.noname-b con 0x7fef740adcb0 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:53.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.457+0000 7fef73fff640 10 monclient(hunting): picked mon.noname-c con 0x7fef7413f910 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:53.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.457+0000 7fef73fff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef740af000 con 0x7fef74071f40 2024-03-29T10:48:53.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.457+0000 7fef73fff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef740ab060 con 0x7fef7413f910 2024-03-29T10:48:53.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.457+0000 7fef73fff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef740aec40 con 0x7fef740adcb0 2024-03-29T10:48:53.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.457+0000 7fef73fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:53.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.458+0000 7fef73fff640 10 monclient(hunting): authenticate will time out at 946.699085s 2024-03-29T10:48:53.444 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.461+0000 7fef72ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fef74071f40 0x7fef7413f200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43829/0 (socket says 172.21.15.5:43829) 2024-03-29T10:48:53.444 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.461+0000 7fef72ffd640 1 -- 172.21.15.5:0/2346794020 learned_addr learned my addr 172.21.15.5:0/2346794020 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:53.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.462+0000 7fef71ffb640 1 -- 172.21.15.5:0/2346794020 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601665378 0 0) 0x7fef740af000 con 0x7fef74071f40 2024-03-29T10:48:53.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.462+0000 7fef71ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:53.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.462+0000 7fef71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:53.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.462+0000 7fef71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:53.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.462+0000 7fef71ffb640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:48:53.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.462+0000 7fef71ffb640 1 -- 172.21.15.5:0/2346794020 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef5c003520 con 0x7fef74071f40 2024-03-29T10:48:53.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 1 -- 172.21.15.5:0/2346794020 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129269109 0 0) 0x7fef5c003520 con 0x7fef74071f40 2024-03-29T10:48:53.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 1 -- 172.21.15.5:0/2346794020 >> v1:172.21.15.5:6790/0 conn(0x7fef740adcb0 legacy=0x7fef74145560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:53.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 1 -- 172.21.15.5:0/2346794020 >> v1:172.21.15.177:6789/0 conn(0x7fef7413f910 legacy=0x7fef74145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:53.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:53.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 1 monclient: found mon.noname-a 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 1 -- 172.21.15.5:0/2346794020 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef740b0030 con 0x7fef74071f40 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 1 -- 172.21.15.5:0/2346794020 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fef64001690 con 0x7fef74071f40 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: dump: 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:53.449 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:53.450 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:53.450 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:53.450 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:53.450 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:53.450 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:53.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 1 -- 172.21.15.5:0/2346794020 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef64001860 con 0x7fef74071f40 2024-03-29T10:48:53.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:53.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 1 -- 172.21.15.5:0/2346794020 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fef64001cb0 con 0x7fef74071f40 2024-03-29T10:48:53.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:53.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.466+0000 7fef71ffb640 10 monclient: dump: 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:53.451 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:53.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.468+0000 7fef73fff640 5 monclient: authenticate success, global_id 4266 2024-03-29T10:48:53.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.468+0000 7fef73fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:53.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.468+0000 7fef73fff640 4 set_mon_vals no callback set 2024-03-29T10:48:53.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.469+0000 7fef73fff640 10 monclient: shutdown 2024-03-29T10:48:53.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.469+0000 7fef73fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:53.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.469+0000 7fef73fff640 1 -- 172.21.15.5:0/2346794020 >> v1:172.21.15.5:6789/0 conn(0x7fef74071f40 legacy=0x7fef7413f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:53.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.469+0000 7fef73fff640 1 -- 172.21.15.5:0/2346794020 shutdown_connections 2024-03-29T10:48:53.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.469+0000 7fef73fff640 1 -- 172.21.15.5:0/2346794020 >> 172.21.15.5:0/2346794020 conn(0x7fef740673e0 msgr2=0x7fef740b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:53.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.469+0000 7fef73fff640 1 -- 172.21.15.5:0/2346794020 shutdown_connections 2024-03-29T10:48:53.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.473+0000 7fef73fff640 1 -- 172.21.15.5:0/2346794020 wait complete. 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.474+0000 7fef73fff640 10 monclient: build_initial_monmap 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.474+0000 7fef73fff640 10 monclient: monmap: 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:48:53.456 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:53.457 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.474+0000 7fef73fff640 1 librados: starting msgr at 2024-03-29T10:48:53.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.474+0000 7fef73fff640 1 librados: starting objecter 2024-03-29T10:48:53.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.474+0000 7fef73fff640 1 Processor -- start 2024-03-29T10:48:53.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.474+0000 7fef73fff640 1 -- start start 2024-03-29T10:48:53.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.474+0000 7fef73fff640 1 librados: setting wanted keys 2024-03-29T10:48:53.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.474+0000 7fef73fff640 1 librados: calling monclient init 2024-03-29T10:48:53.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.474+0000 7fef73fff640 10 monclient: init 2024-03-29T10:48:53.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.475+0000 7fef73fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:53.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.475+0000 7fef73fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef73fff640 10 monclient(hunting): picked mon.noname-b con 0x7fef74071f40 addr v1:172.21.15.177:6789/0 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef73fff640 10 monclient(hunting): picked mon.noname-a con 0x7fef740adcb0 addr v1:172.21.15.5:6789/0 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef73fff640 10 monclient(hunting): picked mon.noname-c con 0x7fef7413f910 addr v1:172.21.15.5:6790/0 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef73fff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7412f350 con 0x7fef740adcb0 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef73fff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef741315a0 con 0x7fef74071f40 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef73fff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef741da470 con 0x7fef7413f910 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef73fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef73fff640 10 monclient(hunting): authenticate will time out at 946.720287s 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef72ffd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fef74071f40 0x7fef741da970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:50327/0 (socket says 172.21.15.5:50327) 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef72ffd640 1 -- 172.21.15.5:0/3184990195 learned_addr learned my addr 172.21.15.5:0/3184990195 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef5b7fe640 10 client.?.objecter ms_handle_connect 0x7fef74071f40 2024-03-29T10:48:53.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.479+0000 7fef5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:53.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320934573 0 0) 0x7fef741315a0 con 0x7fef74071f40 2024-03-29T10:48:53.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:53.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:53.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:53.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:48:53.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef60004eb0 con 0x7fef74071f40 2024-03-29T10:48:53.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 838689993 0 0) 0x7fef60004eb0 con 0x7fef74071f40 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef741315a0 con 0x7fef74071f40 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fef64001b70 con 0x7fef74071f40 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 10 monclient(hunting): dump: 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:53.466 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:53.467 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:53.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:53.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 10 client.?.objecter ms_handle_connect 0x7fef7413f910 2024-03-29T10:48:53.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.482+0000 7fef5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:53.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.483+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 66615301 0 0) 0x7fef741315a0 con 0x7fef74071f40 2024-03-29T10:48:53.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.483+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 >> v1:172.21.15.5:6790/0 conn(0x7fef7413f910 legacy=0x7fef7412ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:53.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.483+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 >> v1:172.21.15.5:6789/0 conn(0x7fef740adcb0 legacy=0x7fef7412e530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:53.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef5b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:53.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef5b7fe640 1 monclient: found mon.b 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef5b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef74130660 con 0x7fef74071f40 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef5b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef5b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef73fff640 5 monclient: authenticate success, global_id 4252 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef73fff640 10 monclient: _renew_subs 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef73fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef73fff640 1 -- 172.21.15.5:0/3184990195 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef741db080 con 0x7fef74071f40 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef73fff640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef73fff640 10 monclient: _renew_subs 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef73fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:48:53.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef73fff640 1 -- 172.21.15.5:0/3184990195 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef74143940 con 0x7fef74071f40 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.484+0000 7fef73fff640 1 librados: init done 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.485+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef64004bf0 con 0x7fef74071f40 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.485+0000 7fef5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.485+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fef64005040 con 0x7fef74071f40 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.485+0000 7fef5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.485+0000 7fef5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.485+0000 7fef5b7fe640 10 monclient: dump: 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:48:53.471 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:48:53.472 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:48:53.472 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:48:53.472 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:48:53.473 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.486+0000 7fef717fa640 4 set_mon_vals no callback set 2024-03-29T10:48:53.473 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.488+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fef640051a0 con 0x7fef74071f40 2024-03-29T10:48:53.473 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.489+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fef64054aa0 con 0x7fef74071f40 2024-03-29T10:48:53.473 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.489+0000 7fef73fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:53.473 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.489+0000 7fef5b7fe640 10 client.4252.objecter ms_dispatch 0x7fef74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:53.473 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.489+0000 7fef5b7fe640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:53.473 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.489+0000 7fef5b7fe640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:53.473 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.489+0000 7fef73fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:53.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.489+0000 7fef73fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:48:53.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.489+0000 7fef73fff640 1 -- 172.21.15.5:0/3184990195 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef741da600 con 0x7fef74071f40 2024-03-29T10:48:53.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.489+0000 7fef5b7fe640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-29T10:48:53.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.489+0000 7fef727fc640 10 monclient: get_auth_request con 0x7fef6006dcb0 auth_method 0 2024-03-29T10:48:53.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.490+0000 7fef5b7fe640 10 client.4252.objecter ms_handle_connect 0x7fef6006dcb0 2024-03-29T10:48:53.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.498+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef64050d40 con 0x7fef74071f40 2024-03-29T10:48:53.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.498+0000 7fef5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:53.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.498+0000 7fef5b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:53.813 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.829+0000 7fef73fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:53.813 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.829+0000 7fef73fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:53.813 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.829+0000 7fef73fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:48:53.813 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.829+0000 7fef73fff640 1 -- 172.21.15.5:0/3184990195 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef7404f730 con 0x7fef74071f40 2024-03-29T10:48:53.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.832+0000 7fef5b7fe640 1 -- 172.21.15.5:0/3184990195 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fef64050ee0 con 0x7fef74071f40 2024-03-29T10:48:53.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.832+0000 7fef5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:53.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.832+0000 7fef5b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:53.816 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:48:53.817 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:53.821 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.838+0000 7fef73fff640 10 librados: watch_flush enter 2024-03-29T10:48:53.821 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.838+0000 7fef73fff640 10 librados: watch_flush exit 2024-03-29T10:48:53.821 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.838+0000 7fef73fff640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-29T10:48:53.821 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.838+0000 7fef73fff640 10 client.4252.objecter successfully canceled tick 2024-03-29T10:48:53.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.838+0000 7fef73fff640 1 -- 172.21.15.5:0/3184990195 >> v1:172.21.15.5:6816/2847500151 conn(0x7fef6006dcb0 legacy=0x7fef60070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:53.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.838+0000 7fef73fff640 10 monclient: shutdown 2024-03-29T10:48:53.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.838+0000 7fef73fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:53.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.838+0000 7fef73fff640 1 -- 172.21.15.5:0/3184990195 >> v1:172.21.15.177:6789/0 conn(0x7fef74071f40 legacy=0x7fef741da970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:53.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.839+0000 7fef73fff640 1 -- 172.21.15.5:0/3184990195 shutdown_connections 2024-03-29T10:48:53.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.839+0000 7fef73fff640 1 -- 172.21.15.5:0/3184990195 >> 172.21.15.5:0/3184990195 conn(0x7fef740673e0 msgr2=0x7fef74069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:53.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.839+0000 7fef73fff640 1 -- 172.21.15.5:0/3184990195 shutdown_connections 2024-03-29T10:48:53.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.839+0000 7fef73fff640 1 -- 172.21.15.5:0/3184990195 wait complete. 2024-03-29T10:48:53.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:48:53.839+0000 7fef73fff640 1 librados: shutdown 2024-03-29T10:48:53.839 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:00.646 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:00.791 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.809+0000 7f8b6359e640 10 monclient: get_monmap_and_config 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.809+0000 7f8b6359e640 10 monclient: build_initial_monmap 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.809+0000 7f8b6359e640 10 monclient: monmap: 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:49:00.810593+0000 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:49:00.810593+0000 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:49:00.792 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:49:00.793 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:00.793 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.811+0000 7f8b6359e640 1 Processor -- start 2024-03-29T10:49:00.793 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.811+0000 7f8b6359e640 1 -- start start 2024-03-29T10:49:00.793 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.811+0000 7f8b6359e640 10 monclient: init 2024-03-29T10:49:00.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.812+0000 7f8b6359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:00.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.812+0000 7f8b6359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:00.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.812+0000 7f8b6359e640 10 monclient(hunting): picked mon.noname-a con 0x7f8b6406fd00 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:00.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.812+0000 7f8b6359e640 10 monclient(hunting): picked mon.noname-c con 0x7f8b640700e0 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:00.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.812+0000 7f8b6359e640 10 monclient(hunting): picked mon.noname-b con 0x7f8b641399f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:00.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.812+0000 7f8b6359e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b64071d50 con 0x7f8b6406fd00 2024-03-29T10:49:00.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.812+0000 7f8b6359e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b641402e0 con 0x7f8b640700e0 2024-03-29T10:49:00.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.812+0000 7f8b6359e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b64071fc0 con 0x7f8b641399f0 2024-03-29T10:49:00.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.812+0000 7f8b6359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:00.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.812+0000 7f8b6359e640 10 monclient(hunting): authenticate will time out at 954.053818s 2024-03-29T10:49:00.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.816+0000 7f8b62d9d640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f8b641399f0 0x7f8b6414b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:39934/0 (socket says 172.21.15.5:39934) 2024-03-29T10:49:00.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.816+0000 7f8b62d9d640 1 -- 172.21.15.5:0/3953041639 learned_addr learned my addr 172.21.15.5:0/3953041639 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:00.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.816+0000 7f8b6159a640 1 -- 172.21.15.5:0/3953041639 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377489914 0 0) 0x7f8b64071fc0 con 0x7f8b641399f0 2024-03-29T10:49:00.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.816+0000 7f8b6159a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:00.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.816+0000 7f8b6159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:00.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.816+0000 7f8b6159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:00.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.816+0000 7f8b6159a640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:49:00.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.816+0000 7f8b6159a640 1 -- 172.21.15.5:0/3953041639 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b4c003520 con 0x7f8b641399f0 2024-03-29T10:49:00.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.817+0000 7f8b6159a640 1 -- 172.21.15.5:0/3953041639 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2787855238 0 0) 0x7f8b4c003520 con 0x7f8b641399f0 2024-03-29T10:49:00.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.817+0000 7f8b6159a640 1 -- 172.21.15.5:0/3953041639 >> v1:172.21.15.177:6789/0 conn(0x7f8b640700e0 legacy=0x7f8b641407c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:00.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.817+0000 7f8b6159a640 1 -- 172.21.15.5:0/3953041639 >> v1:172.21.15.5:6789/0 conn(0x7f8b6406fd00 legacy=0x7f8b640b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:00.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.817+0000 7f8b6159a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:00.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.817+0000 7f8b6159a640 1 monclient: found mon.noname-b 2024-03-29T10:49:00.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.817+0000 7f8b6159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:00.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:49:00.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6159a640 1 -- 172.21.15.5:0/3953041639 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b640722a0 con 0x7f8b641399f0 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6159a640 10 monclient: _finish_auth 0 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6159a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6359e640 5 monclient: authenticate success, global_id 4277 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6159a640 1 -- 172.21.15.5:0/3953041639 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8b5c001a10 con 0x7f8b641399f0 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6159a640 10 monclient: dump: 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:00.803 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.818+0000 7f8b6359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.819+0000 7f8b6159a640 1 -- 172.21.15.5:0/3953041639 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b5c001cb0 con 0x7f8b641399f0 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.819+0000 7f8b6159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.819+0000 7f8b6359e640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.819+0000 7f8b6359e640 4 set_mon_vals no callback set 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.820+0000 7f8b6159a640 1 -- 172.21.15.5:0/3953041639 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8b5c002040 con 0x7f8b641399f0 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.820+0000 7f8b6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.820+0000 7f8b6159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:00.804 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.820+0000 7f8b6159a640 10 monclient: dump: 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.820+0000 7f8b6359e640 10 monclient: shutdown 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.820+0000 7f8b6359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.820+0000 7f8b6359e640 1 -- 172.21.15.5:0/3953041639 >> v1:172.21.15.5:6790/0 conn(0x7f8b641399f0 legacy=0x7f8b6414b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.820+0000 7f8b6359e640 1 -- 172.21.15.5:0/3953041639 shutdown_connections 2024-03-29T10:49:00.805 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.820+0000 7f8b6359e640 1 -- 172.21.15.5:0/3953041639 >> 172.21.15.5:0/3953041639 conn(0x7f8b640672a0 msgr2=0x7f8b640b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:00.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.821+0000 7f8b6359e640 1 -- 172.21.15.5:0/3953041639 shutdown_connections 2024-03-29T10:49:00.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.821+0000 7f8b6359e640 1 -- 172.21.15.5:0/3953041639 wait complete. 2024-03-29T10:49:00.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.822+0000 7f8b6359e640 10 monclient: build_initial_monmap 2024-03-29T10:49:00.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.822+0000 7f8b6359e640 10 monclient: monmap: 2024-03-29T10:49:00.807 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:00.807 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:00.807 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:49:00.807 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.822+0000 7f8b6359e640 1 librados: starting msgr at 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.822+0000 7f8b6359e640 1 librados: starting objecter 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.822+0000 7f8b6359e640 1 Processor -- start 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 1 -- start start 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 1 librados: setting wanted keys 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 1 librados: calling monclient init 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 10 monclient: init 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:00.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:00.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 10 monclient(hunting): picked mon.noname-b con 0x7f8b6406fd00 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:00.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 10 monclient(hunting): picked mon.noname-c con 0x7f8b640700e0 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:00.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 10 monclient(hunting): picked mon.noname-a con 0x7f8b641399f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:00.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6413d4d0 con 0x7f8b641399f0 2024-03-29T10:49:00.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b641d2990 con 0x7f8b6406fd00 2024-03-29T10:49:00.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6413b0a0 con 0x7f8b640700e0 2024-03-29T10:49:00.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:00.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.823+0000 7f8b6359e640 10 monclient(hunting): authenticate will time out at 954.065006s 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b62d9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f8b641399f0 0x7f8b64138b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:41317/0 (socket says 172.21.15.5:41317) 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b62d9d640 1 -- 172.21.15.5:0/2094480922 learned_addr learned my addr 172.21.15.5:0/2094480922 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b61d9b640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f8b640700e0 0x7f8b6413bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:40690/0 (socket says 172.21.15.5:40690) 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 client.?.objecter ms_handle_connect 0x7f8b641399f0 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 client.?.objecter ms_handle_connect 0x7f8b640700e0 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438511274 0 0) 0x7f8b6413d4d0 con 0x7f8b641399f0 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:00.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b44004eb0 con 0x7f8b641399f0 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730307408 0 0) 0x7f8b6413b0a0 con 0x7f8b640700e0 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.827+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b6413d4d0 con 0x7f8b640700e0 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 10 client.?.objecter ms_handle_connect 0x7f8b6406fd00 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 733794865 0 0) 0x7f8b44004eb0 con 0x7f8b641399f0 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b6413b0a0 con 0x7f8b641399f0 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8b5c0019d0 con 0x7f8b641399f0 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:00.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3448264245 0 0) 0x7f8b6413d4d0 con 0x7f8b640700e0 2024-03-29T10:49:00.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b44004eb0 con 0x7f8b640700e0 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8b580017b0 con 0x7f8b640700e0 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:00.818 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1996587536 0 0) 0x7f8b6413b0a0 con 0x7f8b641399f0 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 >> v1:172.21.15.5:6790/0 conn(0x7f8b640700e0 legacy=0x7f8b6413bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 >> v1:172.21.15.177:6789/0 conn(0x7f8b6406fd00 legacy=0x7f8b641d2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.828+0000 7f8b52ffd640 1 monclient: found mon.a 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b52ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b641d35c0 con 0x7f8b641399f0 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b52ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b52ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b6359e640 5 monclient: authenticate success, global_id 4269 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b6359e640 10 monclient: _renew_subs 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b6359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:00.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b6359e640 1 -- 172.21.15.5:0/2094480922 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b64142680 con 0x7f8b641399f0 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b6359e640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b6359e640 10 monclient: _renew_subs 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b6359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b6359e640 1 -- 172.21.15.5:0/2094480922 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6413c400 con 0x7f8b641399f0 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.829+0000 7f8b6359e640 1 librados: init done 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.831+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b5c004e10 con 0x7f8b641399f0 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.831+0000 7f8b52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.831+0000 7f8b53fff640 4 set_mon_vals no callback set 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.831+0000 7f8b50ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.831+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8b5c005120 con 0x7f8b641399f0 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.831+0000 7f8b52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.831+0000 7f8b52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:00.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.831+0000 7f8b52ffd640 10 monclient: dump: 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:00.821 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.832+0000 7f8b50ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:00.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.832+0000 7f8b50ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:00.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.832+0000 7f8b50ff9640 1 -- 172.21.15.5:0/2094480922 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b48005080 con 0x7f8b641399f0 2024-03-29T10:49:00.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.837+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f8b5c01b0f0 con 0x7f8b641399f0 2024-03-29T10:49:00.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.838+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f8b5c084ed0 con 0x7f8b641399f0 2024-03-29T10:49:00.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.838+0000 7f8b6259c640 10 monclient: get_auth_request con 0x7f8b4406df10 auth_method 0 2024-03-29T10:49:00.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.838+0000 7f8b52ffd640 10 client.4269.objecter ms_dispatch 0x7f8b64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:00.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.838+0000 7f8b52ffd640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:00.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.838+0000 7f8b52ffd640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:00.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.839+0000 7f8b52ffd640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T10:49:00.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.839+0000 7f8b52ffd640 10 client.4269.objecter ms_handle_connect 0x7f8b4406df10 2024-03-29T10:49:00.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.843+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b5c0507b0 con 0x7f8b641399f0 2024-03-29T10:49:00.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.843+0000 7f8b52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:00.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:00.843+0000 7f8b52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:01.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.176+0000 7f8b50ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:01.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.176+0000 7f8b50ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:01.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.176+0000 7f8b50ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:01.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.176+0000 7f8b50ff9640 1 -- 172.21.15.5:0/2094480922 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b48005330 con 0x7f8b641399f0 2024-03-29T10:49:01.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.177+0000 7f8b52ffd640 1 -- 172.21.15.5:0/2094480922 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f8b5c0526e0 con 0x7f8b641399f0 2024-03-29T10:49:01.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.177+0000 7f8b52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:01.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.177+0000 7f8b52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:01.160 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:49:01.160 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:01.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.181+0000 7f8b50ff9640 10 librados: watch_flush enter 2024-03-29T10:49:01.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.181+0000 7f8b50ff9640 10 librados: watch_flush exit 2024-03-29T10:49:01.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.181+0000 7f8b50ff9640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T10:49:01.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.181+0000 7f8b50ff9640 10 client.4269.objecter successfully canceled tick 2024-03-29T10:49:01.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.181+0000 7f8b50ff9640 1 -- 172.21.15.5:0/2094480922 >> v1:172.21.15.5:6816/2847500151 conn(0x7f8b4406df10 legacy=0x7f8b440703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:01.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.181+0000 7f8b50ff9640 10 monclient: shutdown 2024-03-29T10:49:01.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.181+0000 7f8b50ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:01.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.181+0000 7f8b50ff9640 1 -- 172.21.15.5:0/2094480922 >> v1:172.21.15.5:6789/0 conn(0x7f8b641399f0 legacy=0x7f8b64138b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:01.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.181+0000 7f8b50ff9640 1 -- 172.21.15.5:0/2094480922 shutdown_connections 2024-03-29T10:49:01.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.181+0000 7f8b50ff9640 1 -- 172.21.15.5:0/2094480922 >> 172.21.15.5:0/2094480922 conn(0x7f8b640672a0 msgr2=0x7f8b640b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:01.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.182+0000 7f8b50ff9640 1 -- 172.21.15.5:0/2094480922 shutdown_connections 2024-03-29T10:49:01.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.182+0000 7f8b50ff9640 1 -- 172.21.15.5:0/2094480922 wait complete. 2024-03-29T10:49:01.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:01.182+0000 7f8b50ff9640 1 librados: shutdown 2024-03-29T10:49:01.182 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:07.989 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:08.139 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.157+0000 7f9c5b3c7640 10 monclient: get_monmap_and_config 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.157+0000 7f9c5b3c7640 10 monclient: build_initial_monmap 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.157+0000 7f9c5b3c7640 10 monclient: monmap: 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:49:08.158590+0000 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:49:08.158590+0000 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:49:08.140 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:08.141 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.158+0000 7f9c5b3c7640 1 Processor -- start 2024-03-29T10:49:08.141 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.159+0000 7f9c5b3c7640 1 -- start start 2024-03-29T10:49:08.141 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.159+0000 7f9c5b3c7640 10 monclient: init 2024-03-29T10:49:08.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c5b3c7640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:08.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c5b3c7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:08.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c5b3c7640 10 monclient(hunting): picked mon.noname-b con 0x7f9c540b0500 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:08.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c5b3c7640 10 monclient(hunting): picked mon.noname-a con 0x7f9c540b08e0 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:08.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c5b3c7640 10 monclient(hunting): picked mon.noname-c con 0x7f9c540aeca0 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:08.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c5b3c7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c54140620 con 0x7f9c540b08e0 2024-03-29T10:49:08.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c5b3c7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5413daf0 con 0x7f9c540aeca0 2024-03-29T10:49:08.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c5b3c7640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c540ae180 con 0x7f9c540b0500 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c5b3c7640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c5b3c7640 10 monclient(hunting): authenticate will time out at 961.401376s 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c59964640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f9c540aeca0 0x7f9c5414b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:42692/0 (socket says 172.21.15.5:42692) 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.160+0000 7f9c59964640 1 -- 172.21.15.5:0/3450115336 learned_addr learned my addr 172.21.15.5:0/3450115336 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.161+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789197904 0 0) 0x7f9c5413daf0 con 0x7f9c540aeca0 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.161+0000 7f9c4bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.161+0000 7f9c4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.161+0000 7f9c4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.161+0000 7f9c4bfff640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.161+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c40003520 con 0x7f9c540aeca0 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.161+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087765596 0 0) 0x7f9c540ae180 con 0x7f9c540b0500 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.161+0000 7f9c4bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.161+0000 7f9c4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:08.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.161+0000 7f9c4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5413daf0 con 0x7f9c540b0500 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2242637925 0 0) 0x7f9c40003520 con 0x7f9c540aeca0 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 >> v1:172.21.15.5:6790/0 conn(0x7f9c540b0500 legacy=0x7f9c5413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 >> v1:172.21.15.5:6789/0 conn(0x7f9c540b08e0 legacy=0x7f9c5413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 1 monclient: found mon.noname-c 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54072510 con 0x7f9c540aeca0 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: _finish_auth 0 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:08.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9c50001a10 con 0x7f9c540aeca0 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: dump: 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:08.151 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c5b3c7640 5 monclient: authenticate success, global_id 4258 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c5b3c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c50001b70 con 0x7f9c540aeca0 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 1 -- 172.21.15.5:0/3450115336 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9c50001fc0 con 0x7f9c540aeca0 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c4bfff640 10 monclient: dump: 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:08.152 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c5b3c7640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.162+0000 7f9c5b3c7640 4 set_mon_vals no callback set 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.163+0000 7f9c5b3c7640 10 monclient: shutdown 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.163+0000 7f9c5b3c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.163+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3450115336 >> v1:172.21.15.177:6789/0 conn(0x7f9c540aeca0 legacy=0x7f9c5414b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.163+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3450115336 shutdown_connections 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.164+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3450115336 >> 172.21.15.5:0/3450115336 conn(0x7f9c5406b660 msgr2=0x7f9c5406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.164+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3450115336 shutdown_connections 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.164+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3450115336 wait complete. 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.165+0000 7f9c5b3c7640 10 monclient: build_initial_monmap 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.165+0000 7f9c5b3c7640 10 monclient: monmap: 2024-03-29T10:49:08.153 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:08.154 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:08.154 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:49:08.154 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:49:08.154 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:08.154 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:08.154 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:08.154 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:49:08.154 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:49:08.154 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:08.154 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.165+0000 7f9c5b3c7640 1 librados: starting msgr at 2024-03-29T10:49:08.155 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.165+0000 7f9c5b3c7640 1 librados: starting objecter 2024-03-29T10:49:08.155 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.166+0000 7f9c5b3c7640 1 Processor -- start 2024-03-29T10:49:08.155 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.166+0000 7f9c5b3c7640 1 -- start start 2024-03-29T10:49:08.155 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.166+0000 7f9c5b3c7640 1 librados: setting wanted keys 2024-03-29T10:49:08.155 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.166+0000 7f9c5b3c7640 1 librados: calling monclient init 2024-03-29T10:49:08.155 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.166+0000 7f9c5b3c7640 10 monclient: init 2024-03-29T10:49:08.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c5b3c7640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:08.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c5b3c7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:08.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c5b3c7640 10 monclient(hunting): picked mon.noname-a con 0x7f9c540aeca0 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c5b3c7640 10 monclient(hunting): picked mon.noname-c con 0x7f9c540b0500 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c5b3c7640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5411b8d0 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c5b3c7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5411d590 con 0x7f9c540aeca0 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c5b3c7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5411efa0 con 0x7f9c5411b8d0 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c5b3c7640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5411d7f0 con 0x7f9c540b0500 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c5b3c7640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c5b3c7640 10 monclient(hunting): authenticate will time out at 961.408230s 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c58962640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f9c540b0500 0x7f9c5413a880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:59498/0 (socket says 172.21.15.5:59498) 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c58962640 1 -- 172.21.15.5:0/3820519584 learned_addr learned my addr 172.21.15.5:0/3820519584 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c49ffb640 10 client.?.objecter ms_handle_connect 0x7f9c540b0500 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c49ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c49ffb640 10 client.?.objecter ms_handle_connect 0x7f9c5411b8d0 2024-03-29T10:49:08.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.167+0000 7f9c49ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 client.?.objecter ms_handle_connect 0x7f9c540aeca0 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135922558 0 0) 0x7f9c5411efa0 con 0x7f9c5411b8d0 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c38004eb0 con 0x7f9c5411b8d0 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614077619 0 0) 0x7f9c5411d590 con 0x7f9c540aeca0 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5411efa0 con 0x7f9c540aeca0 2024-03-29T10:49:08.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 911511052 0 0) 0x7f9c38004eb0 con 0x7f9c5411b8d0 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c5411d590 con 0x7f9c5411b8d0 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9c50001e80 con 0x7f9c5411b8d0 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): dump: 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:08.164 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397090045 0 0) 0x7f9c5411d7f0 con 0x7f9c540b0500 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.168+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c380057f0 con 0x7f9c540b0500 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2058031423 0 0) 0x7f9c5411d590 con 0x7f9c5411b8d0 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 >> v1:172.21.15.5:6790/0 conn(0x7f9c540b0500 legacy=0x7f9c5413a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 >> v1:172.21.15.5:6789/0 conn(0x7f9c540aeca0 legacy=0x7f9c5411b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 1 monclient: found mon.b 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:08.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5411c940 con 0x7f9c5411b8d0 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c5b3c7640 5 monclient: authenticate success, global_id 4261 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c5b3c7640 10 monclient: _renew_subs 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c5b3c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3820519584 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5411dd10 con 0x7f9c5411b8d0 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c5b3c7640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c5b3c7640 10 monclient: _renew_subs 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c5b3c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3820519584 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5411cdd0 con 0x7f9c5411b8d0 2024-03-29T10:49:08.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.169+0000 7f9c5b3c7640 1 librados: init done 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.170+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c50004e10 con 0x7f9c5411b8d0 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.170+0000 7f9c49ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.170+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9c50005120 con 0x7f9c5411b8d0 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.170+0000 7f9c49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.170+0000 7f9c49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.170+0000 7f9c49ffb640 10 monclient: dump: 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:08.167 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.170+0000 7f9c4b7fe640 4 set_mon_vals no callback set 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.171+0000 7f9c5b3c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.172+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f9c5001b0f0 con 0x7f9c5411b8d0 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.173+0000 7f9c5b3c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.173+0000 7f9c5b3c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.173+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3820519584 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c540ad0c0 con 0x7f9c5411b8d0 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.173+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f9c50084dd0 con 0x7f9c5411b8d0 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.173+0000 7f9c49ffb640 10 client.4261.objecter ms_dispatch 0x7f9c54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.173+0000 7f9c49ffb640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.173+0000 7f9c49ffb640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.173+0000 7f9c49ffb640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.173+0000 7f9c59163640 10 monclient: get_auth_request con 0x7f9c3806df50 auth_method 0 2024-03-29T10:49:08.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.174+0000 7f9c49ffb640 10 client.4261.objecter ms_handle_connect 0x7f9c3806df50 2024-03-29T10:49:08.169 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.179+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c50050710 con 0x7f9c5411b8d0 2024-03-29T10:49:08.169 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.179+0000 7f9c49ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:08.169 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.179+0000 7f9c49ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:08.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.583+0000 7f9c5b3c7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:08.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.583+0000 7f9c5b3c7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:08.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.583+0000 7f9c5b3c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:49:08.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.583+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3820519584 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c5411ede0 con 0x7f9c5411b8d0 2024-03-29T10:49:08.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.585+0000 7f9c49ffb640 1 -- 172.21.15.5:0/3820519584 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f9c50084480 con 0x7f9c5411b8d0 2024-03-29T10:49:08.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.585+0000 7f9c49ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:08.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.585+0000 7f9c49ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:08.567 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:49:08.567 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:08.570 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.588+0000 7f9c5b3c7640 10 librados: watch_flush enter 2024-03-29T10:49:08.570 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.588+0000 7f9c5b3c7640 10 librados: watch_flush exit 2024-03-29T10:49:08.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.588+0000 7f9c5b3c7640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T10:49:08.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.588+0000 7f9c5b3c7640 10 client.4261.objecter successfully canceled tick 2024-03-29T10:49:08.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.588+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3820519584 >> v1:172.21.15.5:6816/2847500151 conn(0x7f9c3806df50 legacy=0x7f9c38070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:08.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.588+0000 7f9c5b3c7640 10 monclient: shutdown 2024-03-29T10:49:08.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.588+0000 7f9c5b3c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:08.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.588+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3820519584 >> v1:172.21.15.177:6789/0 conn(0x7f9c5411b8d0 legacy=0x7f9c541de8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:08.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.589+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3820519584 shutdown_connections 2024-03-29T10:49:08.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.589+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3820519584 >> 172.21.15.5:0/3820519584 conn(0x7f9c5406b660 msgr2=0x7f9c5406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:08.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.589+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3820519584 shutdown_connections 2024-03-29T10:49:08.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.589+0000 7f9c5b3c7640 1 -- 172.21.15.5:0/3820519584 wait complete. 2024-03-29T10:49:08.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:08.589+0000 7f9c5b3c7640 1 librados: shutdown 2024-03-29T10:49:08.587 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:15.395 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:15.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.550+0000 7fceef4e2640 10 monclient: get_monmap_and_config 2024-03-29T10:49:15.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.550+0000 7fceef4e2640 10 monclient: build_initial_monmap 2024-03-29T10:49:15.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.550+0000 7fceef4e2640 10 monclient: monmap: 2024-03-29T10:49:15.533 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:15.533 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:15.533 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:49:15.551523+0000 2024-03-29T10:49:15.533 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:49:15.551523+0000 2024-03-29T10:49:15.533 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:15.533 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:15.533 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:15.533 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:49:15.533 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:49:15.533 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:15.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.551+0000 7fceef4e2640 1 Processor -- start 2024-03-29T10:49:15.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.551+0000 7fceef4e2640 1 -- start start 2024-03-29T10:49:15.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.551+0000 7fceef4e2640 10 monclient: init 2024-03-29T10:49:15.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceef4e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:15.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceef4e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceef4e2640 10 monclient(hunting): picked mon.noname-c con 0x7fcee81457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceef4e2640 10 monclient(hunting): picked mon.noname-a con 0x7fcee813f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceef4e2640 10 monclient(hunting): picked mon.noname-b con 0x7fcee80a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceef4e2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee80aaef0 con 0x7fcee813f650 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceef4e2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee8145bd0 con 0x7fcee81457f0 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceef4e2640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee8145eb0 con 0x7fcee80a9d70 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceef4e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceef4e2640 10 monclient(hunting): authenticate will time out at 968.793682s 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceeda7f640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fcee80a9d70 0x7fcee814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:50322/0 (socket says 172.21.15.5:50322) 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.552+0000 7fceeda7f640 1 -- 172.21.15.5:0/821640952 learned_addr learned my addr 172.21.15.5:0/821640952 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:15.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.553+0000 7fcedffff640 1 -- 172.21.15.5:0/821640952 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903288552 0 0) 0x7fcee8145eb0 con 0x7fcee80a9d70 2024-03-29T10:49:15.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.553+0000 7fcedffff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:15.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.553+0000 7fcedffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:15.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.553+0000 7fcedffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:15.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.553+0000 7fcedffff640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:49:15.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.553+0000 7fcedffff640 1 -- 172.21.15.5:0/821640952 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced4003520 con 0x7fcee80a9d70 2024-03-29T10:49:15.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.553+0000 7fcedffff640 1 -- 172.21.15.5:0/821640952 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 593260089 0 0) 0x7fced4003520 con 0x7fcee80a9d70 2024-03-29T10:49:15.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.553+0000 7fcedffff640 1 -- 172.21.15.5:0/821640952 >> v1:172.21.15.177:6789/0 conn(0x7fcee81457f0 legacy=0x7fcee813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:15.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.553+0000 7fcedffff640 1 -- 172.21.15.5:0/821640952 >> v1:172.21.15.5:6789/0 conn(0x7fcee813f650 legacy=0x7fcee80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:15.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:15.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 1 monclient: found mon.noname-b 2024-03-29T10:49:15.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:15.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:49:15.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 1 -- 172.21.15.5:0/821640952 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee8146340 con 0x7fcee80a9d70 2024-03-29T10:49:15.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: _finish_auth 0 2024-03-29T10:49:15.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:15.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:15.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 1 -- 172.21.15.5:0/821640952 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fcee00019f0 con 0x7fcee80a9d70 2024-03-29T10:49:15.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: dump: 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:15.546 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fceef4e2640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:49:15.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fceef4e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:15.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 1 -- 172.21.15.5:0/821640952 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcee0001c80 con 0x7fcee80a9d70 2024-03-29T10:49:15.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:15.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 1 -- 172.21.15.5:0/821640952 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fcee0002050 con 0x7fcee80a9d70 2024-03-29T10:49:15.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:15.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:15.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fcedffff640 10 monclient: dump: 2024-03-29T10:49:15.547 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:15.547 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:15.547 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fceef4e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.555+0000 7fceef4e2640 4 set_mon_vals no callback set 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.556+0000 7fceef4e2640 10 monclient: shutdown 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.556+0000 7fceef4e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:15.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.556+0000 7fceef4e2640 1 -- 172.21.15.5:0/821640952 >> v1:172.21.15.5:6790/0 conn(0x7fcee80a9d70 legacy=0x7fcee814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.556+0000 7fceef4e2640 1 -- 172.21.15.5:0/821640952 shutdown_connections 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.556+0000 7fceef4e2640 1 -- 172.21.15.5:0/821640952 >> 172.21.15.5:0/821640952 conn(0x7fcee8066a00 msgr2=0x7fcee8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.557+0000 7fceef4e2640 1 -- 172.21.15.5:0/821640952 shutdown_connections 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.557+0000 7fceef4e2640 1 -- 172.21.15.5:0/821640952 wait complete. 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.558+0000 7fceef4e2640 10 monclient: build_initial_monmap 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.558+0000 7fceef4e2640 10 monclient: monmap: 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:49:15.549 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.558+0000 7fceef4e2640 1 librados: starting msgr at 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.558+0000 7fceef4e2640 1 librados: starting objecter 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.558+0000 7fceef4e2640 1 Processor -- start 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.558+0000 7fceef4e2640 1 -- start start 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.558+0000 7fceef4e2640 1 librados: setting wanted keys 2024-03-29T10:49:15.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.558+0000 7fceef4e2640 1 librados: calling monclient init 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.558+0000 7fceef4e2640 10 monclient: init 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.559+0000 7fceef4e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.559+0000 7fceef4e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.559+0000 7fceef4e2640 10 monclient(hunting): picked mon.noname-b con 0x7fcee80a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.559+0000 7fceed27e640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fcee80a9d70 0x7fcee81dbae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:39138/0 (socket says 172.21.15.5:39138) 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.559+0000 7fceeca7d640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fcee813f650 0x7fcee813b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:50514/0 (socket says 172.21.15.5:50514) 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.559+0000 7fceeca7d640 1 -- 172.21.15.5:0/2046412958 learned_addr learned my addr 172.21.15.5:0/2046412958 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.559+0000 7fceef4e2640 10 monclient(hunting): picked mon.noname-c con 0x7fcee813f650 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.559+0000 7fceddffb640 10 client.?.objecter ms_handle_connect 0x7fcee813f650 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.559+0000 7fceddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:15.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.560+0000 7fceef4e2640 10 monclient(hunting): picked mon.noname-a con 0x7fcee81457f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.560+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee813bfd0 con 0x7fcee81457f0 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.560+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee81df6b0 con 0x7fcee80a9d70 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.560+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee81dc1f0 con 0x7fcee813f650 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.560+0000 7fceef4e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.560+0000 7fceef4e2640 10 monclient(hunting): authenticate will time out at 968.801995s 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 10 client.?.objecter ms_handle_connect 0x7fcee80a9d70 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287039182 0 0) 0x7fcee81dc1f0 con 0x7fcee813f650 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:15.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcecc004eb0 con 0x7fcee813f650 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658066714 0 0) 0x7fcee81df6b0 con 0x7fcee80a9d70 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.561+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee81dc1f0 con 0x7fcee80a9d70 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 458076689 0 0) 0x7fcee81dc1f0 con 0x7fcee80a9d70 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcee81df6b0 con 0x7fcee80a9d70 2024-03-29T10:49:15.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1094833030 0 0) 0x7fcecc004eb0 con 0x7fcee813f650 2024-03-29T10:49:15.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcee81dc1f0 con 0x7fcee813f650 2024-03-29T10:49:15.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fcee4001890 con 0x7fcee80a9d70 2024-03-29T10:49:15.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:15.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:15.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient(hunting): dump: 2024-03-29T10:49:15.554 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:15.554 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:15.554 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:15.554 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:15.554 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fced8001960 con 0x7fcee813f650 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient(hunting): dump: 2024-03-29T10:49:15.555 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3136975545 0 0) 0x7fcee81dc1f0 con 0x7fcee813f650 2024-03-29T10:49:15.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 >> v1:172.21.15.177:6789/0 conn(0x7fcee80a9d70 legacy=0x7fcee81dbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:15.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 >> v1:172.21.15.5:6789/0 conn(0x7fcee81457f0 legacy=0x7fcee813b8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:15.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:15.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 monclient: found mon.c 2024-03-29T10:49:15.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:15.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:15.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee81dfbd0 con 0x7fcee813f650 2024-03-29T10:49:15.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:15.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:15.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:15.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceef4e2640 5 monclient: authenticate success, global_id 4292 2024-03-29T10:49:15.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceef4e2640 10 monclient: _renew_subs 2024-03-29T10:49:15.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceef4e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:15.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.562+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee81dfdc0 con 0x7fcee813f650 2024-03-29T10:49:15.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.563+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fced8001b40 con 0x7fcee813f650 2024-03-29T10:49:15.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.563+0000 7fceddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:15.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.563+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fced8003a40 con 0x7fcee813f650 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.563+0000 7fceddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.563+0000 7fceddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.563+0000 7fceddffb640 10 monclient: dump: 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:15.563 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.563+0000 7fcedf7fe640 4 set_mon_vals no callback set 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.564+0000 7fceef4e2640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.564+0000 7fceef4e2640 10 monclient: _renew_subs 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.564+0000 7fceef4e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.564+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee80b0fb0 con 0x7fcee813f650 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.564+0000 7fceef4e2640 1 librados: init done 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.565+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fced8003c20 con 0x7fcee813f650 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.565+0000 7fceef4e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.565+0000 7fceef4e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.565+0000 7fceef4e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.565+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee81df8f0 con 0x7fcee813f650 2024-03-29T10:49:15.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.566+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fced8084be0 con 0x7fcee813f650 2024-03-29T10:49:15.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.566+0000 7fceddffb640 10 client.4292.objecter ms_dispatch 0x7fcee8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:15.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.566+0000 7fceddffb640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:15.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.566+0000 7fceddffb640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:15.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.566+0000 7fceed27e640 10 monclient: get_auth_request con 0x7fcecc06ded0 auth_method 0 2024-03-29T10:49:15.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.566+0000 7fceddffb640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T10:49:15.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.567+0000 7fceddffb640 10 client.4292.objecter ms_handle_connect 0x7fcecc06ded0 2024-03-29T10:49:15.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.572+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcee81df8f0 con 0x7fcee813f650 2024-03-29T10:49:15.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.572+0000 7fceddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:15.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.572+0000 7fceddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:15.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.888+0000 7fceef4e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:15.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.888+0000 7fceef4e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:15.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.888+0000 7fceef4e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:15.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.888+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcee81df8f0 con 0x7fcee813f650 2024-03-29T10:49:15.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.889+0000 7fceddffb640 1 -- 172.21.15.5:0/2046412958 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fcee81df8f0 con 0x7fcee813f650 2024-03-29T10:49:15.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.889+0000 7fceddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:15.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.889+0000 7fceddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:15.872 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:49:15.872 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:15.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 10 librados: watch_flush enter 2024-03-29T10:49:15.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 10 librados: watch_flush exit 2024-03-29T10:49:15.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T10:49:15.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 10 client.4292.objecter successfully canceled tick 2024-03-29T10:49:15.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 >> v1:172.21.15.5:6816/2847500151 conn(0x7fcecc06ded0 legacy=0x7fcecc070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:15.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 10 monclient: shutdown 2024-03-29T10:49:15.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:15.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 >> v1:172.21.15.5:6790/0 conn(0x7fcee813f650 legacy=0x7fcee813b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:15.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 shutdown_connections 2024-03-29T10:49:15.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 >> 172.21.15.5:0/2046412958 conn(0x7fcee8066a00 msgr2=0x7fcee80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:15.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 shutdown_connections 2024-03-29T10:49:15.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 1 -- 172.21.15.5:0/2046412958 wait complete. 2024-03-29T10:49:15.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:15.892+0000 7fceef4e2640 1 librados: shutdown 2024-03-29T10:49:15.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:22.696 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:22.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.844+0000 7fc7d14ae640 10 monclient: get_monmap_and_config 2024-03-29T10:49:22.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.844+0000 7fc7d14ae640 10 monclient: build_initial_monmap 2024-03-29T10:49:22.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.844+0000 7fc7d14ae640 10 monclient: monmap: 2024-03-29T10:49:22.826 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:22.826 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:22.827 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:49:22.845643+0000 2024-03-29T10:49:22.827 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:49:22.845643+0000 2024-03-29T10:49:22.827 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:22.827 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:22.827 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:22.827 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:49:22.827 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:49:22.827 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:22.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.845+0000 7fc7d14ae640 1 Processor -- start 2024-03-29T10:49:22.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.845+0000 7fc7d14ae640 1 -- start start 2024-03-29T10:49:22.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.845+0000 7fc7d14ae640 10 monclient: init 2024-03-29T10:49:22.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.846+0000 7fc7d14ae640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:22.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.846+0000 7fc7d14ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:22.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.847+0000 7fc7d14ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc7cc1457f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:22.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.847+0000 7fc7d14ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc7cc13f650 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:22.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.847+0000 7fc7d14ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc7cc0a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:22.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.847+0000 7fc7d14ae640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7cc0aaef0 con 0x7fc7cc1457f0 2024-03-29T10:49:22.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.847+0000 7fc7d14ae640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7cc145bd0 con 0x7fc7cc13f650 2024-03-29T10:49:22.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.847+0000 7fc7d14ae640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7cc145eb0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.847+0000 7fc7d14ae640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:22.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.847+0000 7fc7d14ae640 10 monclient(hunting): authenticate will time out at 976.088704s 2024-03-29T10:49:22.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7c3fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fc7cc13f650 0x7fc7cc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:34384/0 (socket says 172.21.15.5:34384) 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7c3fff640 1 -- 172.21.15.5:0/4134007134 learned_addr learned my addr 172.21.15.5:0/4134007134 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959273890 0 0) 0x7fc7cc145bd0 con 0x7fc7cc13f650 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b8003520 con 0x7fc7cc13f650 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558112693 0 0) 0x7fc7cc145eb0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7cc145bd0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 860022703 0 0) 0x7fc7b8003520 con 0x7fc7cc13f650 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 >> v1:172.21.15.5:6790/0 conn(0x7fc7cc0a9d70 legacy=0x7fc7cc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.848+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 >> v1:172.21.15.5:6789/0 conn(0x7fc7cc1457f0 legacy=0x7fc7cc13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 1 monclient: found mon.noname-c 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc146340 con 0x7fc7cc13f650 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc7b4001690 con 0x7fc7cc13f650 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7ca7fc640 10 monclient: dump: 2024-03-29T10:49:22.833 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7d14ae640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:49:22.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.849+0000 7fc7d14ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:22.835 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.850+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7b4001870 con 0x7fc7cc13f650 2024-03-29T10:49:22.835 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.850+0000 7fc7ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.850+0000 7fc7ca7fc640 1 -- 172.21.15.5:0/4134007134 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc7b4001c40 con 0x7fc7cc13f650 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.850+0000 7fc7ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.850+0000 7fc7ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.850+0000 7fc7ca7fc640 10 monclient: dump: 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:22.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.850+0000 7fc7d14ae640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.850+0000 7fc7d14ae640 4 set_mon_vals no callback set 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.851+0000 7fc7d14ae640 10 monclient: shutdown 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.851+0000 7fc7d14ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.851+0000 7fc7d14ae640 1 -- 172.21.15.5:0/4134007134 >> v1:172.21.15.177:6789/0 conn(0x7fc7cc13f650 legacy=0x7fc7cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.852+0000 7fc7d14ae640 1 -- 172.21.15.5:0/4134007134 shutdown_connections 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.852+0000 7fc7d14ae640 1 -- 172.21.15.5:0/4134007134 >> 172.21.15.5:0/4134007134 conn(0x7fc7cc066a00 msgr2=0x7fc7cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.852+0000 7fc7d14ae640 1 -- 172.21.15.5:0/4134007134 shutdown_connections 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.852+0000 7fc7d14ae640 1 -- 172.21.15.5:0/4134007134 wait complete. 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.853+0000 7fc7d14ae640 10 monclient: build_initial_monmap 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.853+0000 7fc7d14ae640 10 monclient: monmap: 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:22.837 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:22.838 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:22.838 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:49:22.838 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:49:22.838 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:22.840 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.855+0000 7fc7d14ae640 1 librados: starting msgr at 2024-03-29T10:49:22.840 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.855+0000 7fc7d14ae640 1 librados: starting objecter 2024-03-29T10:49:22.840 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.855+0000 7fc7d14ae640 1 Processor -- start 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.855+0000 7fc7d14ae640 1 -- start start 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.855+0000 7fc7d14ae640 1 librados: setting wanted keys 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.855+0000 7fc7d14ae640 1 librados: calling monclient init 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.855+0000 7fc7d14ae640 10 monclient: init 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7d14ae640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7d14ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7d14ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc7cc0a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7d14ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc7cc13f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7d14ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc7cc1457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7d14ae640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7cc137790 con 0x7fc7cc13f650 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7d14ae640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7cc141a60 con 0x7fc7cc1457f0 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7d14ae640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7cc1422c0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7d14ae640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7cb7fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fc7cc1457f0 0x7fc7cc137080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:35200/0 (socket says 172.21.15.5:35200) 2024-03-29T10:49:22.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7cb7fe640 1 -- 172.21.15.5:0/2476904736 learned_addr learned my addr 172.21.15.5:0/2476904736 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7d14ae640 10 monclient(hunting): authenticate will time out at 976.097516s 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7c37fe640 10 client.?.objecter ms_handle_connect 0x7fc7cc1457f0 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205038127 0 0) 0x7fc7cc141a60 con 0x7fc7cc1457f0 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7c37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7c37fe640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.856+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ac004eb0 con 0x7fc7cc1457f0 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 10 client.?.objecter ms_handle_connect 0x7fc7cc0a9d70 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1678580101 0 0) 0x7fc7ac004eb0 con 0x7fc7cc1457f0 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7cc141a60 con 0x7fc7cc1457f0 2024-03-29T10:49:22.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc7c4001d10 con 0x7fc7cc1457f0 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 10 monclient(hunting): dump: 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:22.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356421802 0 0) 0x7fc7cc1422c0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.857+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ac004eb0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.858+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2000506430 0 0) 0x7fc7ac004eb0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.858+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7cc1422c0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.858+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc7bc001730 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.858+0000 7fc7c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.858+0000 7fc7c37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.858+0000 7fc7c37fe640 10 monclient(hunting): dump: 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:22.844 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:22.845 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:22.845 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:22.845 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:22.845 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:22.845 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:22.845 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.859+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1503807985 0 0) 0x7fc7cc1422c0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.859+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 >> v1:172.21.15.177:6789/0 conn(0x7fc7cc1457f0 legacy=0x7fc7cc137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.859+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 >> v1:172.21.15.5:6789/0 conn(0x7fc7cc13f650 legacy=0x7fc7cc136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.859+0000 7fc7c37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7c37fe640 1 monclient: found mon.c 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7c37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc137a10 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7c37fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7c37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7c37fe640 10 client.?.objecter ms_handle_connect 0x7fc7cc13f650 2024-03-29T10:49:22.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7d14ae640 5 monclient: authenticate success, global_id 4298 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7d14ae640 10 monclient: _renew_subs 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7d14ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7d14ae640 1 -- 172.21.15.5:0/2476904736 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7cc1d7580 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7d14ae640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7d14ae640 10 monclient: _renew_subs 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7d14ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.860+0000 7fc7d14ae640 1 -- 172.21.15.5:0/2476904736 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7cc1429c0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.861+0000 7fc7d14ae640 1 librados: init done 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.861+0000 7fc7c37fe640 10 client.4298.objecter resend_mon_ops 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.861+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7bc001890 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.861+0000 7fc7c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.861+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc7bc001d60 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.861+0000 7fc7c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:22.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.861+0000 7fc7c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.861+0000 7fc7c37fe640 10 monclient: dump: 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.861+0000 7fc7c9ffb640 4 set_mon_vals no callback set 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.862+0000 7fc7d14ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.862+0000 7fc7d14ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:22.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.862+0000 7fc7d14ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:22.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.862+0000 7fc7d14ae640 1 -- 172.21.15.5:0/2476904736 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b0005080 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.864+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fc7bc003790 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.864+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fc7bc0836e0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.864+0000 7fc7c37fe640 10 client.4298.objecter ms_dispatch 0x7fc7cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:22.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.864+0000 7fc7c37fe640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:22.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.864+0000 7fc7c37fe640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:22.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.865+0000 7fc7c37fe640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T10:49:22.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.865+0000 7fc7c3fff640 10 monclient: get_auth_request con 0x7fc7ac06de80 auth_method 0 2024-03-29T10:49:22.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.865+0000 7fc7c37fe640 10 client.4298.objecter ms_handle_connect 0x7fc7ac06de80 2024-03-29T10:49:22.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.870+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7bc04efc0 con 0x7fc7cc0a9d70 2024-03-29T10:49:22.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.870+0000 7fc7c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:22.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:22.870+0000 7fc7c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:23.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.183+0000 7fc7d14ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:23.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.183+0000 7fc7d14ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:23.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.183+0000 7fc7d14ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:23.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.183+0000 7fc7d14ae640 1 -- 172.21.15.5:0/2476904736 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7b0005330 con 0x7fc7cc0a9d70 2024-03-29T10:49:23.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.184+0000 7fc7c37fe640 1 -- 172.21.15.5:0/2476904736 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fc7bc050ef0 con 0x7fc7cc0a9d70 2024-03-29T10:49:23.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.184+0000 7fc7c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:23.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.184+0000 7fc7c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:23.166 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:49:23.167 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:23.169 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.187+0000 7fc7d14ae640 10 librados: watch_flush enter 2024-03-29T10:49:23.169 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.187+0000 7fc7d14ae640 10 librados: watch_flush exit 2024-03-29T10:49:23.169 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.187+0000 7fc7d14ae640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T10:49:23.170 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.187+0000 7fc7d14ae640 10 client.4298.objecter successfully canceled tick 2024-03-29T10:49:23.170 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.188+0000 7fc7d14ae640 1 -- 172.21.15.5:0/2476904736 >> v1:172.21.15.5:6816/2847500151 conn(0x7fc7ac06de80 legacy=0x7fc7ac070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:23.170 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.188+0000 7fc7d14ae640 10 monclient: shutdown 2024-03-29T10:49:23.170 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.188+0000 7fc7d14ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:23.170 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.188+0000 7fc7d14ae640 1 -- 172.21.15.5:0/2476904736 >> v1:172.21.15.5:6790/0 conn(0x7fc7cc0a9d70 legacy=0x7fc7cc1d6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:23.171 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.188+0000 7fc7d14ae640 1 -- 172.21.15.5:0/2476904736 shutdown_connections 2024-03-29T10:49:23.171 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.188+0000 7fc7d14ae640 1 -- 172.21.15.5:0/2476904736 >> 172.21.15.5:0/2476904736 conn(0x7fc7cc066a00 msgr2=0x7fc7cc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:23.171 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.189+0000 7fc7d14ae640 1 -- 172.21.15.5:0/2476904736 shutdown_connections 2024-03-29T10:49:23.171 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.189+0000 7fc7d14ae640 1 -- 172.21.15.5:0/2476904736 wait complete. 2024-03-29T10:49:23.171 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:23.189+0000 7fc7d14ae640 1 librados: shutdown 2024-03-29T10:49:23.185 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:29.992 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:30.144 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.160+0000 7efe691f6640 10 monclient: get_monmap_and_config 2024-03-29T10:49:30.144 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.160+0000 7efe691f6640 10 monclient: build_initial_monmap 2024-03-29T10:49:30.144 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.160+0000 7efe691f6640 10 monclient: monmap: 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:49:30.161844+0000 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:49:30.161844+0000 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:30.145 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.162+0000 7efe691f6640 1 Processor -- start 2024-03-29T10:49:30.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.165+0000 7efe691f6640 1 -- start start 2024-03-29T10:49:30.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.165+0000 7efe691f6640 10 monclient: init 2024-03-29T10:49:30.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.166+0000 7efe691f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:30.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.166+0000 7efe691f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:30.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.166+0000 7efe691f6640 10 monclient(hunting): picked mon.noname-c con 0x7efe5c029eb0 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:30.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.166+0000 7efe691f6640 10 monclient(hunting): picked mon.noname-a con 0x7efe5c02b0a0 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:30.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.166+0000 7efe691f6640 10 monclient(hunting): picked mon.noname-b con 0x7efe5c0c9570 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:30.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.166+0000 7efe691f6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c02a290 con 0x7efe5c02b0a0 2024-03-29T10:49:30.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.166+0000 7efe691f6640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c0331c0 con 0x7efe5c029eb0 2024-03-29T10:49:30.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.166+0000 7efe691f6640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c02c100 con 0x7efe5c0c9570 2024-03-29T10:49:30.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.166+0000 7efe691f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:30.150 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.166+0000 7efe691f6640 10 monclient(hunting): authenticate will time out at 983.407903s 2024-03-29T10:49:30.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.171+0000 7efe63fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7efe5c029eb0 0x7efe5c0cf120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:55003/0 (socket says 172.21.15.5:55003) 2024-03-29T10:49:30.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.171+0000 7efe63fff640 1 -- 172.21.15.5:0/1611700359 learned_addr learned my addr 172.21.15.5:0/1611700359 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:30.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 1 -- 172.21.15.5:0/1611700359 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973769447 0 0) 0x7efe5c02a290 con 0x7efe5c02b0a0 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 1 -- 172.21.15.5:0/1611700359 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe54003520 con 0x7efe5c02b0a0 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 1 -- 172.21.15.5:0/1611700359 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877295589 0 0) 0x7efe5c0331c0 con 0x7efe5c029eb0 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.174+0000 7efe62ffd640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 1 -- 172.21.15.5:0/1611700359 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5c02a290 con 0x7efe5c029eb0 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 1 -- 172.21.15.5:0/1611700359 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 133140134 0 0) 0x7efe54003520 con 0x7efe5c02b0a0 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 1 -- 172.21.15.5:0/1611700359 >> v1:172.21.15.5:6790/0 conn(0x7efe5c0c9570 legacy=0x7efe5c0d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:30.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 1 -- 172.21.15.5:0/1611700359 >> v1:172.21.15.177:6789/0 conn(0x7efe5c029eb0 legacy=0x7efe5c0cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 1 monclient: found mon.noname-a 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 1 -- 172.21.15.5:0/1611700359 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c02a440 con 0x7efe5c02b0a0 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 1 -- 172.21.15.5:0/1611700359 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efe580015c0 con 0x7efe5c02b0a0 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 10 monclient: dump: 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:30.160 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe691f6640 5 monclient: authenticate success, global_id 4275 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe691f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 1 -- 172.21.15.5:0/1611700359 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe580017a0 con 0x7efe5c02b0a0 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe691f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.175+0000 7efe691f6640 4 set_mon_vals no callback set 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.176+0000 7efe691f6640 10 monclient: shutdown 2024-03-29T10:49:30.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.176+0000 7efe691f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:30.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.176+0000 7efe691f6640 1 -- 172.21.15.5:0/1611700359 >> v1:172.21.15.5:6789/0 conn(0x7efe5c02b0a0 legacy=0x7efe5c0c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:30.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.176+0000 7efe691f6640 1 -- 172.21.15.5:0/1611700359 shutdown_connections 2024-03-29T10:49:30.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.176+0000 7efe691f6640 1 -- 172.21.15.5:0/1611700359 >> 172.21.15.5:0/1611700359 conn(0x7efe5c0058f0 msgr2=0x7efe5c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:30.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.177+0000 7efe691f6640 1 -- 172.21.15.5:0/1611700359 shutdown_connections 2024-03-29T10:49:30.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.177+0000 7efe691f6640 1 -- 172.21.15.5:0/1611700359 wait complete. 2024-03-29T10:49:30.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.178+0000 7efe691f6640 10 monclient: build_initial_monmap 2024-03-29T10:49:30.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.178+0000 7efe691f6640 10 monclient: monmap: 2024-03-29T10:49:30.165 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.178+0000 7efe691f6640 1 librados: starting msgr at 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.178+0000 7efe691f6640 1 librados: starting objecter 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.178+0000 7efe691f6640 1 Processor -- start 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.181+0000 7efe691f6640 1 -- start start 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.181+0000 7efe691f6640 1 librados: setting wanted keys 2024-03-29T10:49:30.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.181+0000 7efe691f6640 1 librados: calling monclient init 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.181+0000 7efe691f6640 10 monclient: init 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.182+0000 7efe691f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.182+0000 7efe691f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.182+0000 7efe691f6640 10 monclient(hunting): picked mon.noname-b con 0x7efe5c029eb0 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.182+0000 7efe691f6640 10 monclient(hunting): picked mon.noname-a con 0x7efe5c02b0a0 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.182+0000 7efe691f6640 10 monclient(hunting): picked mon.noname-c con 0x7efe5c0c9570 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.182+0000 7efe691f6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c0cd490 con 0x7efe5c02b0a0 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.182+0000 7efe691f6640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c02da20 con 0x7efe5c029eb0 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.182+0000 7efe691f6640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c02caa0 con 0x7efe5c0c9570 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.182+0000 7efe691f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:30.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.182+0000 7efe691f6640 10 monclient(hunting): authenticate will time out at 983.423743s 2024-03-29T10:49:30.170 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.188+0000 7efe637fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7efe5c02b0a0 0x7efe5c02df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51721/0 (socket says 172.21.15.5:51721) 2024-03-29T10:49:30.170 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.188+0000 7efe637fe640 1 -- 172.21.15.5:0/1524640148 learned_addr learned my addr 172.21.15.5:0/1524640148 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:30.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.189+0000 7efe60ff9640 10 client.?.objecter ms_handle_connect 0x7efe5c029eb0 2024-03-29T10:49:30.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.189+0000 7efe60ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:30.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.189+0000 7efe60ff9640 10 client.?.objecter ms_handle_connect 0x7efe5c0c9570 2024-03-29T10:49:30.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.189+0000 7efe60ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:30.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.189+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831471279 0 0) 0x7efe5c02da20 con 0x7efe5c029eb0 2024-03-29T10:49:30.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.189+0000 7efe60ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:30.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.189+0000 7efe60ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:30.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.189+0000 7efe60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:30.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.189+0000 7efe60ff9640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:49:30.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.189+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe48004eb0 con 0x7efe5c029eb0 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 10 client.?.objecter ms_handle_connect 0x7efe5c02b0a0 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696614624 0 0) 0x7efe5c02caa0 con 0x7efe5c0c9570 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5c02da20 con 0x7efe5c0c9570 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999042721 0 0) 0x7efe5c0cd490 con 0x7efe5c02b0a0 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:30.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5c02caa0 con 0x7efe5c02b0a0 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 398156819 0 0) 0x7efe48004eb0 con 0x7efe5c029eb0 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.190+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe5c0cd490 con 0x7efe5c029eb0 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.191+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efe500013b0 con 0x7efe5c029eb0 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.191+0000 7efe60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.191+0000 7efe60ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.191+0000 7efe60ff9640 10 monclient(hunting): dump: 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:30.178 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.191+0000 7efe60ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3959640617 0 0) 0x7efe5c02da20 con 0x7efe5c0c9570 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe48004eb0 con 0x7efe5c0c9570 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efe6404fdd0 con 0x7efe5c0c9570 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 10 monclient(hunting): dump: 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:30.179 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 997125544 0 0) 0x7efe5c0cd490 con 0x7efe5c029eb0 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 >> v1:172.21.15.5:6790/0 conn(0x7efe5c0c9570 legacy=0x7efe5c1853e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 >> v1:172.21.15.5:6789/0 conn(0x7efe5c02b0a0 legacy=0x7efe5c02df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 1 monclient: found mon.b 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:49:30.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c0c0550 con 0x7efe5c029eb0 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 10 monclient: _finish_auth 0 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe691f6640 5 monclient: authenticate success, global_id 4279 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe691f6640 10 monclient: _renew_subs 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe691f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe691f6640 1 -- 172.21.15.5:0/1524640148 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c0c0820 con 0x7efe5c029eb0 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe691f6640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe691f6640 10 monclient: _renew_subs 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe691f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe691f6640 1 -- 172.21.15.5:0/1524640148 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5c02d690 con 0x7efe5c029eb0 2024-03-29T10:49:30.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.192+0000 7efe691f6640 1 librados: init done 2024-03-29T10:49:30.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.194+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe50001c90 con 0x7efe5c029eb0 2024-03-29T10:49:30.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.194+0000 7efe60ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:30.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.194+0000 7efe61ffb640 4 set_mon_vals no callback set 2024-03-29T10:49:30.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.195+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efe500016f0 con 0x7efe5c029eb0 2024-03-29T10:49:30.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.195+0000 7efe60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:30.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.195+0000 7efe60ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:30.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.195+0000 7efe60ff9640 10 monclient: dump: 2024-03-29T10:49:30.184 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:30.184 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:30.184 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:30.184 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.196+0000 7efe691f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.197+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7efe5001a900 con 0x7efe5c029eb0 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.198+0000 7efe691f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.198+0000 7efe691f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.198+0000 7efe691f6640 1 -- 172.21.15.5:0/1524640148 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe44005080 con 0x7efe5c029eb0 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.198+0000 7efe637fe640 10 monclient: get_auth_request con 0x7efe4806de70 auth_method 0 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.198+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7efe500849e0 con 0x7efe5c029eb0 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.198+0000 7efe60ff9640 10 client.4279.objecter ms_dispatch 0x7efe64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:30.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.198+0000 7efe60ff9640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:30.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.198+0000 7efe60ff9640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:30.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.198+0000 7efe60ff9640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T10:49:30.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.199+0000 7efe60ff9640 10 client.4279.objecter ms_handle_connect 0x7efe4806de70 2024-03-29T10:49:30.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.206+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe500502c0 con 0x7efe5c029eb0 2024-03-29T10:49:30.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.206+0000 7efe60ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:30.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.206+0000 7efe60ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:30.516 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.534+0000 7efe691f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:30.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.534+0000 7efe691f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:30.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.534+0000 7efe691f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:49:30.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.534+0000 7efe691f6640 1 -- 172.21.15.5:0/1524640148 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe44005330 con 0x7efe5c029eb0 2024-03-29T10:49:30.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.538+0000 7efe60ff9640 1 -- 172.21.15.5:0/1524640148 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7efe500521f0 con 0x7efe5c029eb0 2024-03-29T10:49:30.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.538+0000 7efe60ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:30.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.538+0000 7efe60ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:30.520 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:49:30.520 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:30.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.541+0000 7efe691f6640 10 librados: watch_flush enter 2024-03-29T10:49:30.524 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.542+0000 7efe691f6640 10 librados: watch_flush exit 2024-03-29T10:49:30.524 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.542+0000 7efe691f6640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T10:49:30.524 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.542+0000 7efe691f6640 10 client.4279.objecter successfully canceled tick 2024-03-29T10:49:30.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.543+0000 7efe691f6640 1 -- 172.21.15.5:0/1524640148 >> v1:172.21.15.5:6816/2847500151 conn(0x7efe4806de70 legacy=0x7efe48070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:30.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.543+0000 7efe691f6640 10 monclient: shutdown 2024-03-29T10:49:30.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.543+0000 7efe691f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:30.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.543+0000 7efe691f6640 1 -- 172.21.15.5:0/1524640148 >> v1:172.21.15.177:6789/0 conn(0x7efe5c029eb0 legacy=0x7efe5c0bf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:30.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.544+0000 7efe691f6640 1 -- 172.21.15.5:0/1524640148 shutdown_connections 2024-03-29T10:49:30.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.544+0000 7efe691f6640 1 -- 172.21.15.5:0/1524640148 >> 172.21.15.5:0/1524640148 conn(0x7efe5c0058f0 msgr2=0x7efe5c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:30.527 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.545+0000 7efe691f6640 1 -- 172.21.15.5:0/1524640148 shutdown_connections 2024-03-29T10:49:30.528 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.545+0000 7efe691f6640 1 -- 172.21.15.5:0/1524640148 wait complete. 2024-03-29T10:49:30.528 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:30.546+0000 7efe691f6640 1 librados: shutdown 2024-03-29T10:49:30.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:37.354 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:37.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.536+0000 7fddeef65640 10 monclient: get_monmap_and_config 2024-03-29T10:49:37.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.536+0000 7fddeef65640 10 monclient: build_initial_monmap 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.536+0000 7fddeef65640 10 monclient: monmap: 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:49:37.537713+0000 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:49:37.537713+0000 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:49:37.519 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:37.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.539+0000 7fddeef65640 1 Processor -- start 2024-03-29T10:49:37.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.539+0000 7fddeef65640 1 -- start start 2024-03-29T10:49:37.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.539+0000 7fddeef65640 10 monclient: init 2024-03-29T10:49:37.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.540+0000 7fddeef65640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:37.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.540+0000 7fddeef65640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:37.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.540+0000 7fddeef65640 10 monclient(hunting): picked mon.noname-c con 0x7fdde806ee40 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:37.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.540+0000 7fddeef65640 10 monclient(hunting): picked mon.noname-a con 0x7fdde80aeda0 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:37.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.540+0000 7fddeef65640 10 monclient(hunting): picked mon.noname-b con 0x7fdde813c470 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:37.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.540+0000 7fddeef65640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde806fdb0 con 0x7fdde80aeda0 2024-03-29T10:49:37.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.540+0000 7fddeef65640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde8071090 con 0x7fdde806ee40 2024-03-29T10:49:37.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.540+0000 7fddeef65640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde8071370 con 0x7fdde813c470 2024-03-29T10:49:37.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.540+0000 7fddeef65640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:37.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.540+0000 7fddeef65640 10 monclient(hunting): authenticate will time out at 990.781239s 2024-03-29T10:49:37.524 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.542+0000 7fdded502640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fdde813c470 0x7fdde814a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:52123/0 (socket says 172.21.15.5:52123) 2024-03-29T10:49:37.524 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.542+0000 7fdded502640 1 -- 172.21.15.5:0/3195066029 learned_addr learned my addr 172.21.15.5:0/3195066029 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:37.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.543+0000 7fdde77fe640 1 -- 172.21.15.5:0/3195066029 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705856405 0 0) 0x7fdde8071370 con 0x7fdde813c470 2024-03-29T10:49:37.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.543+0000 7fdde77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:37.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.543+0000 7fdde77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:37.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.543+0000 7fdde77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:37.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.543+0000 7fdde77fe640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:49:37.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.543+0000 7fdde77fe640 1 -- 172.21.15.5:0/3195066029 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd0003520 con 0x7fdde813c470 2024-03-29T10:49:37.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.544+0000 7fdde77fe640 1 -- 172.21.15.5:0/3195066029 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3581143944 0 0) 0x7fddd0003520 con 0x7fdde813c470 2024-03-29T10:49:37.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.544+0000 7fdde77fe640 1 -- 172.21.15.5:0/3195066029 >> v1:172.21.15.177:6789/0 conn(0x7fdde806ee40 legacy=0x7fdde813bd60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:37.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.545+0000 7fdde77fe640 1 -- 172.21.15.5:0/3195066029 >> v1:172.21.15.5:6789/0 conn(0x7fdde80aeda0 legacy=0x7fdde8071790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:37.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:37.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 1 monclient: found mon.noname-b 2024-03-29T10:49:37.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:37.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:49:37.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 1 -- 172.21.15.5:0/3195066029 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde8070220 con 0x7fdde813c470 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 1 -- 172.21.15.5:0/3195066029 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fdddc001690 con 0x7fdde813c470 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fdde77fe640 10 monclient: dump: 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:37.531 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fddeef65640 5 monclient: authenticate success, global_id 4307 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.547+0000 7fddeef65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.548+0000 7fdde77fe640 1 -- 172.21.15.5:0/3195066029 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdddc001860 con 0x7fdde813c470 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.548+0000 7fdde77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.548+0000 7fdde77fe640 1 -- 172.21.15.5:0/3195066029 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fdddc001cb0 con 0x7fdde813c470 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.548+0000 7fdde77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.548+0000 7fdde77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.548+0000 7fdde77fe640 10 monclient: dump: 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:37.532 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:37.533 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:37.533 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:37.533 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:37.533 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:37.533 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:37.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.549+0000 7fddeef65640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:37.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.549+0000 7fddeef65640 4 set_mon_vals no callback set 2024-03-29T10:49:37.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.549+0000 7fddeef65640 10 monclient: shutdown 2024-03-29T10:49:37.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.549+0000 7fddeef65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:37.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.549+0000 7fddeef65640 1 -- 172.21.15.5:0/3195066029 >> v1:172.21.15.5:6790/0 conn(0x7fdde813c470 legacy=0x7fdde814a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:37.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.550+0000 7fddeef65640 1 -- 172.21.15.5:0/3195066029 shutdown_connections 2024-03-29T10:49:37.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.550+0000 7fddeef65640 1 -- 172.21.15.5:0/3195066029 >> 172.21.15.5:0/3195066029 conn(0x7fdde8067470 msgr2=0x7fdde80b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:37.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.552+0000 7fddeef65640 1 -- 172.21.15.5:0/3195066029 shutdown_connections 2024-03-29T10:49:37.538 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.552+0000 7fddeef65640 1 -- 172.21.15.5:0/3195066029 wait complete. 2024-03-29T10:49:37.538 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.553+0000 7fddeef65640 10 monclient: build_initial_monmap 2024-03-29T10:49:37.538 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.553+0000 7fddeef65640 10 monclient: monmap: 2024-03-29T10:49:37.538 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:37.538 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:37.538 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:49:37.538 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:49:37.538 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.554+0000 7fddeef65640 1 librados: starting msgr at 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.554+0000 7fddeef65640 1 librados: starting objecter 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.554+0000 7fddeef65640 1 Processor -- start 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.554+0000 7fddeef65640 1 -- start start 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.554+0000 7fddeef65640 1 librados: setting wanted keys 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.554+0000 7fddeef65640 1 librados: calling monclient init 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.554+0000 7fddeef65640 10 monclient: init 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.555+0000 7fddeef65640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.555+0000 7fddeef65640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:37.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.555+0000 7fddeef65640 10 monclient(hunting): picked mon.noname-c con 0x7fdde806ee40 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:37.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.555+0000 7fddeef65640 10 monclient(hunting): picked mon.noname-a con 0x7fdde80aeda0 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:37.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.555+0000 7fddeef65640 10 monclient(hunting): picked mon.noname-b con 0x7fdde813c470 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:37.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.555+0000 7fddeef65640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde813e890 con 0x7fdde80aeda0 2024-03-29T10:49:37.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.555+0000 7fddeef65640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde813ed20 con 0x7fdde813c470 2024-03-29T10:49:37.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.555+0000 7fddeef65640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde80ae230 con 0x7fdde806ee40 2024-03-29T10:49:37.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.555+0000 7fddeef65640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:37.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.555+0000 7fddeef65640 10 monclient(hunting): authenticate will time out at 990.796490s 2024-03-29T10:49:37.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.559+0000 7fddecd01640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fdde806ee40 0x7fdde8142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:52237/0 (socket says 172.21.15.5:52237) 2024-03-29T10:49:37.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.559+0000 7fdded502640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fdde813c470 0x7fdde81deed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:46763/0 (socket says 172.21.15.5:46763) 2024-03-29T10:49:37.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.559+0000 7fdded502640 1 -- 172.21.15.5:0/3647449163 learned_addr learned my addr 172.21.15.5:0/3647449163 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:37.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.560+0000 7fdde57fa640 10 client.?.objecter ms_handle_connect 0x7fdde806ee40 2024-03-29T10:49:37.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.561+0000 7fdde57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:37.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.561+0000 7fdde57fa640 10 client.?.objecter ms_handle_connect 0x7fdde813c470 2024-03-29T10:49:37.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.561+0000 7fdde57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:37.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.561+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922065095 0 0) 0x7fdde80ae230 con 0x7fdde806ee40 2024-03-29T10:49:37.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.561+0000 7fdde57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:37.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.561+0000 7fdde57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:37.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.561+0000 7fdde57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:37.544 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.561+0000 7fdde57fa640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:49:37.544 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.561+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd4004eb0 con 0x7fdde806ee40 2024-03-29T10:49:37.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.563+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062357503 0 0) 0x7fdde813ed20 con 0x7fdde813c470 2024-03-29T10:49:37.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.563+0000 7fdde57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:37.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.563+0000 7fdde57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:37.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.563+0000 7fdde57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:37.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.563+0000 7fdde57fa640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:49:37.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.563+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde80ae230 con 0x7fdde813c470 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 10 client.?.objecter ms_handle_connect 0x7fdde80aeda0 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 695958641 0 0) 0x7fddd4004eb0 con 0x7fdde806ee40 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdde813ed20 con 0x7fdde806ee40 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fddd8001870 con 0x7fdde806ee40 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 10 monclient(hunting): dump: 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:37.550 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269575385 0 0) 0x7fdde813e890 con 0x7fdde80aeda0 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.564+0000 7fdde57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.565+0000 7fdde57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.565+0000 7fdde57fa640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.565+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd4004eb0 con 0x7fdde80aeda0 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.565+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 911337827 0 0) 0x7fdde80ae230 con 0x7fdde813c470 2024-03-29T10:49:37.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.565+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdde813e890 con 0x7fdde813c470 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.565+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fdddc001b70 con 0x7fdde813c470 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.565+0000 7fdde57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.565+0000 7fdde57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.565+0000 7fdde57fa640 10 monclient(hunting): dump: 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:37.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.566+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2326490162 0 0) 0x7fddd4004eb0 con 0x7fdde80aeda0 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.566+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdde80ae230 con 0x7fdde80aeda0 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.566+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1605189971 0 0) 0x7fdde813ed20 con 0x7fdde806ee40 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.566+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 >> v1:172.21.15.177:6789/0 conn(0x7fdde813c470 legacy=0x7fdde81deed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.566+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 >> v1:172.21.15.5:6789/0 conn(0x7fdde80aeda0 legacy=0x7fdde80adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.566+0000 7fdde57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.566+0000 7fdde57fa640 1 monclient: found mon.c 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.566+0000 7fdde57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.566+0000 7fdde57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.566+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde80ae3c0 con 0x7fdde806ee40 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fdde57fa640 10 monclient: _finish_auth 0 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fdde57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fdde57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fddeef65640 5 monclient: authenticate success, global_id 4310 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fddeef65640 10 monclient: _renew_subs 2024-03-29T10:49:37.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fddeef65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:37.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fddeef65640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde813d740 con 0x7fdde806ee40 2024-03-29T10:49:37.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fddeef65640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:37.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fddeef65640 10 monclient: _renew_subs 2024-03-29T10:49:37.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fddeef65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:37.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fddeef65640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde813e170 con 0x7fdde806ee40 2024-03-29T10:49:37.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.567+0000 7fddeef65640 1 librados: init done 2024-03-29T10:49:37.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.571+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddd8001d80 con 0x7fdde806ee40 2024-03-29T10:49:37.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.571+0000 7fdde57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:37.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.571+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fddd80049e0 con 0x7fdde806ee40 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.571+0000 7fdde57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.572+0000 7fdde57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.572+0000 7fdde57fa640 10 monclient: dump: 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:37.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.572+0000 7fdde67fc640 4 set_mon_vals no callback set 2024-03-29T10:49:37.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.573+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fddd801a9b0 con 0x7fdde806ee40 2024-03-29T10:49:37.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.573+0000 7fddeef65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:37.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.573+0000 7fddeef65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:37.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.573+0000 7fddeef65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:37.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.573+0000 7fddeef65640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc8005080 con 0x7fdde806ee40 2024-03-29T10:49:37.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.575+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fddd80848f0 con 0x7fdde806ee40 2024-03-29T10:49:37.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.575+0000 7fdde57fa640 10 client.4310.objecter ms_dispatch 0x7fdde8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:37.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.575+0000 7fdde57fa640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:37.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.575+0000 7fdde57fa640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:37.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.575+0000 7fdde57fa640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T10:49:37.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.575+0000 7fdde7fff640 10 monclient: get_auth_request con 0x7fddd406e250 auth_method 0 2024-03-29T10:49:37.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.577+0000 7fdde57fa640 10 client.4310.objecter ms_handle_connect 0x7fddd406e250 2024-03-29T10:49:37.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.583+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fddd8050230 con 0x7fdde806ee40 2024-03-29T10:49:37.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.583+0000 7fdde57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:37.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.583+0000 7fdde57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:37.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.926+0000 7fddeef65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:37.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.926+0000 7fddeef65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:37.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.926+0000 7fddeef65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:49:37.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.926+0000 7fddeef65640 1 -- 172.21.15.5:0/3647449163 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddc8005330 con 0x7fdde806ee40 2024-03-29T10:49:37.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.930+0000 7fdde57fa640 1 -- 172.21.15.5:0/3647449163 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fddd8052160 con 0x7fdde806ee40 2024-03-29T10:49:37.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.930+0000 7fdde57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:37.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.930+0000 7fdde57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:37.912 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:49:37.912 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:37.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.933+0000 7fddeef65640 10 librados: watch_flush enter 2024-03-29T10:49:37.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.935+0000 7fddeef65640 10 librados: watch_flush exit 2024-03-29T10:49:37.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.935+0000 7fddeef65640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T10:49:37.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.935+0000 7fddeef65640 10 client.4310.objecter successfully canceled tick 2024-03-29T10:49:37.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.935+0000 7fddeef65640 1 -- 172.21.15.5:0/3647449163 >> v1:172.21.15.5:6816/2847500151 conn(0x7fddd406e250 legacy=0x7fddd4070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:37.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.935+0000 7fddeef65640 10 monclient: shutdown 2024-03-29T10:49:37.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.935+0000 7fddeef65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:37.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.935+0000 7fddeef65640 1 -- 172.21.15.5:0/3647449163 >> v1:172.21.15.5:6790/0 conn(0x7fdde806ee40 legacy=0x7fdde8142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:37.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.936+0000 7fddeef65640 1 -- 172.21.15.5:0/3647449163 shutdown_connections 2024-03-29T10:49:37.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.936+0000 7fddeef65640 1 -- 172.21.15.5:0/3647449163 >> 172.21.15.5:0/3647449163 conn(0x7fdde8067470 msgr2=0x7fdde813b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:37.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.937+0000 7fddeef65640 1 -- 172.21.15.5:0/3647449163 shutdown_connections 2024-03-29T10:49:37.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.937+0000 7fddeef65640 1 -- 172.21.15.5:0/3647449163 wait complete. 2024-03-29T10:49:37.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:37.937+0000 7fddeef65640 1 librados: shutdown 2024-03-29T10:49:37.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:44.745 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:44.898 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.916+0000 7f1879e01640 10 monclient: get_monmap_and_config 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.916+0000 7f1879e01640 10 monclient: build_initial_monmap 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.916+0000 7f1879e01640 10 monclient: monmap: 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:49:44.917738+0000 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:49:44.917738+0000 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:49:44.899 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:44.901 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.919+0000 7f1879e01640 1 Processor -- start 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 1 -- start start 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 10 monclient: init 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 10 monclient(hunting): picked mon.noname-b con 0x7f186c0389e0 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 10 monclient(hunting): picked mon.noname-c con 0x7f186c038dc0 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 10 monclient(hunting): picked mon.noname-a con 0x7f186c030660 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c0c54c0 con 0x7f186c030660 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c036430 con 0x7f186c038dc0 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c039840 con 0x7f186c0389e0 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:44.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.920+0000 7f1879e01640 10 monclient(hunting): authenticate will time out at 998.161997s 2024-03-29T10:49:44.906 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.924+0000 7f1879600640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f186c030660 0x7f186c0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:37893/0 (socket says 172.21.15.5:37893) 2024-03-29T10:49:44.906 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.924+0000 7f1879600640 1 -- 172.21.15.5:0/1187808728 learned_addr learned my addr 172.21.15.5:0/1187808728 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:44.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880395805 0 0) 0x7f186c0c54c0 con 0x7f186c030660 2024-03-29T10:49:44.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:44.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:44.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:44.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1864003520 con 0x7f186c030660 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249869047 0 0) 0x7f186c039840 con 0x7f186c0389e0 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f186c0c54c0 con 0x7f186c0389e0 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616511132 0 0) 0x7f186c036430 con 0x7f186c038dc0 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:49:44.911 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.926+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f186c039840 con 0x7f186c038dc0 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2830680120 0 0) 0x7f1864003520 con 0x7f186c030660 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 >> v1:172.21.15.5:6790/0 conn(0x7f186c0389e0 legacy=0x7f186c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 >> v1:172.21.15.177:6789/0 conn(0x7f186c038dc0 legacy=0x7f186c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 1 monclient: found mon.noname-a 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c035690 con 0x7f186c030660 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f1860001690 con 0x7f186c030660 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:44.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 10 monclient: dump: 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f1879e01640 5 monclient: authenticate success, global_id 4284 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f1879e01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18600017f0 con 0x7f186c030660 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f18737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:44.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f1879e01640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.927+0000 7f1879e01640 4 set_mon_vals no callback set 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.928+0000 7f18737fe640 1 -- 172.21.15.5:0/1187808728 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f1860001c00 con 0x7f186c030660 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.928+0000 7f18737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.928+0000 7f18737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.928+0000 7f18737fe640 10 monclient: dump: 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:44.914 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:44.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.929+0000 7f1879e01640 10 monclient: shutdown 2024-03-29T10:49:44.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.929+0000 7f1879e01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:44.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.929+0000 7f1879e01640 1 -- 172.21.15.5:0/1187808728 >> v1:172.21.15.5:6789/0 conn(0x7f186c030660 legacy=0x7f186c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:44.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.932+0000 7f1879e01640 1 -- 172.21.15.5:0/1187808728 shutdown_connections 2024-03-29T10:49:44.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.932+0000 7f1879e01640 1 -- 172.21.15.5:0/1187808728 >> 172.21.15.5:0/1187808728 conn(0x7f186c005c00 msgr2=0x7f186c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:44.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.932+0000 7f1879e01640 1 -- 172.21.15.5:0/1187808728 shutdown_connections 2024-03-29T10:49:44.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.932+0000 7f1879e01640 1 -- 172.21.15.5:0/1187808728 wait complete. 2024-03-29T10:49:44.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.933+0000 7f1879e01640 10 monclient: build_initial_monmap 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.933+0000 7f1879e01640 10 monclient: monmap: 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.933+0000 7f1879e01640 1 librados: starting msgr at 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.933+0000 7f1879e01640 1 librados: starting objecter 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.933+0000 7f1879e01640 1 Processor -- start 2024-03-29T10:49:44.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.934+0000 7f1879e01640 1 -- start start 2024-03-29T10:49:44.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.934+0000 7f1879e01640 1 librados: setting wanted keys 2024-03-29T10:49:44.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.934+0000 7f1879e01640 1 librados: calling monclient init 2024-03-29T10:49:44.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.934+0000 7f1879e01640 10 monclient: init 2024-03-29T10:49:44.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.935+0000 7f1879e01640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:44.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.935+0000 7f1879e01640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:44.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.935+0000 7f1879e01640 10 monclient(hunting): picked mon.noname-c con 0x7f186c030660 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:44.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.935+0000 7f1879e01640 10 monclient(hunting): picked mon.noname-b con 0x7f186c0389e0 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:44.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.935+0000 7f1879e01640 10 monclient(hunting): picked mon.noname-a con 0x7f186c02ea60 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:44.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.935+0000 7f1879e01640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c00b680 con 0x7f186c02ea60 2024-03-29T10:49:44.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.935+0000 7f1879e01640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c00b960 con 0x7f186c0389e0 2024-03-29T10:49:44.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.935+0000 7f1879e01640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c00bc40 con 0x7f186c030660 2024-03-29T10:49:44.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.935+0000 7f1879e01640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:44.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.935+0000 7f1879e01640 10 monclient(hunting): authenticate will time out at 998.176919s 2024-03-29T10:49:44.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.939+0000 7f1878dff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f186c030660 0x7f186c0099e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:45469/0 (socket says 172.21.15.5:45469) 2024-03-29T10:49:44.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.939+0000 7f1879600640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f186c02ea60 0x7f186c16c030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:37995/0 (socket says 172.21.15.5:37995) 2024-03-29T10:49:44.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.939+0000 7f1879600640 1 -- 172.21.15.5:0/724293257 learned_addr learned my addr 172.21.15.5:0/724293257 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:44.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 client.?.objecter ms_handle_connect 0x7f186c02ea60 2024-03-29T10:49:44.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045960055 0 0) 0x7f186c00b680 con 0x7f186c02ea60 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1858004eb0 con 0x7f186c02ea60 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 client.?.objecter ms_handle_connect 0x7f186c030660 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459161365 0 0) 0x7f186c00bc40 con 0x7f186c030660 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.940+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18580051d0 con 0x7f186c030660 2024-03-29T10:49:44.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.941+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 83004374 0 0) 0x7f1858004eb0 con 0x7f186c02ea60 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.941+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f186c00bc40 con 0x7f186c02ea60 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.941+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f1860001bc0 con 0x7f186c02ea60 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.941+0000 7f18717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.941+0000 7f18717fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.941+0000 7f18717fa640 10 monclient(hunting): dump: 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:44.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.941+0000 7f18717fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:44.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 260232236 0 0) 0x7f186c00bc40 con 0x7f186c02ea60 2024-03-29T10:49:44.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 >> v1:172.21.15.5:6790/0 conn(0x7f186c030660 legacy=0x7f186c0099e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 >> v1:172.21.15.177:6789/0 conn(0x7f186c0389e0 legacy=0x7f186c00c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 10 monclient: _finish_hunting 0 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 1 monclient: found mon.a 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c00a190 con 0x7f186c02ea60 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 10 monclient: _finish_auth 0 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 10 monclient: _check_auth_tickets 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f18717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f1879e01640 5 monclient: authenticate success, global_id 4287 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f1879e01640 10 monclient: _renew_subs 2024-03-29T10:49:44.930 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f1879e01640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f1879e01640 1 -- 172.21.15.5:0/724293257 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186c00c770 con 0x7f186c02ea60 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f1879e01640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f1879e01640 10 monclient: _renew_subs 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f1879e01640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f1879e01640 1 -- 172.21.15.5:0/724293257 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f186c00ab00 con 0x7f186c02ea60 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.945+0000 7f1879e01640 1 librados: init done 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.946+0000 7f185effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.946+0000 7f185effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.946+0000 7f185effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.946+0000 7f185effd640 1 -- 172.21.15.5:0/724293257 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186c007b90 con 0x7f186c02ea60 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.946+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1860004a90 con 0x7f186c02ea60 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.946+0000 7f18717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.946+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f1860001250 con 0x7f186c02ea60 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.946+0000 7f18717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:44.931 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.946+0000 7f18717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.946+0000 7f18717fa640 10 monclient: dump: 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:44.932 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.947+0000 7f1872ffd640 4 set_mon_vals no callback set 2024-03-29T10:49:44.933 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.949+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f1860004ee0 con 0x7f186c02ea60 2024-03-29T10:49:44.933 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.949+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f1860054930 con 0x7f186c02ea60 2024-03-29T10:49:44.933 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.949+0000 7f18717fa640 10 client.4287.objecter ms_dispatch 0x7f1874000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:44.933 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.949+0000 7f18717fa640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:44.933 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.949+0000 7f18717fa640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:44.933 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.949+0000 7f18717fa640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-29T10:49:44.933 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.950+0000 7f1878dff640 10 monclient: get_auth_request con 0x7f185806ded0 auth_method 0 2024-03-29T10:49:44.934 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.950+0000 7f18717fa640 10 client.4287.objecter ms_handle_connect 0x7f185806ded0 2024-03-29T10:49:44.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.956+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1860050bd0 con 0x7f186c02ea60 2024-03-29T10:49:44.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.956+0000 7f18717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:44.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:44.956+0000 7f18717fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:45.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.291+0000 7f185effd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:45.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.291+0000 7f185effd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:45.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.291+0000 7f185effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:45.274 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.291+0000 7f185effd640 1 -- 172.21.15.5:0/724293257 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f186c005120 con 0x7f186c02ea60 2024-03-29T10:49:45.274 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:49:45.274 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:45.275 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.292+0000 7f18717fa640 1 -- 172.21.15.5:0/724293257 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f1860050bd0 con 0x7f186c02ea60 2024-03-29T10:49:45.275 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.292+0000 7f18717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:45.275 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.292+0000 7f18717fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:45.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.297+0000 7f185effd640 10 librados: watch_flush enter 2024-03-29T10:49:45.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.299+0000 7f185effd640 10 librados: watch_flush exit 2024-03-29T10:49:45.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.299+0000 7f185effd640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-29T10:49:45.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.299+0000 7f185effd640 10 client.4287.objecter successfully canceled tick 2024-03-29T10:49:45.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.300+0000 7f185effd640 1 -- 172.21.15.5:0/724293257 >> v1:172.21.15.5:6816/2847500151 conn(0x7f185806ded0 legacy=0x7f1858070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:45.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.300+0000 7f185effd640 10 monclient: shutdown 2024-03-29T10:49:45.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.300+0000 7f185effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:45.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.300+0000 7f185effd640 1 -- 172.21.15.5:0/724293257 >> v1:172.21.15.5:6789/0 conn(0x7f186c02ea60 legacy=0x7f186c16c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:45.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.300+0000 7f185effd640 1 -- 172.21.15.5:0/724293257 shutdown_connections 2024-03-29T10:49:45.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.300+0000 7f185effd640 1 -- 172.21.15.5:0/724293257 >> 172.21.15.5:0/724293257 conn(0x7f186c005c00 msgr2=0x7f186c03ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:45.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.302+0000 7f185effd640 1 -- 172.21.15.5:0/724293257 shutdown_connections 2024-03-29T10:49:45.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.302+0000 7f185effd640 1 -- 172.21.15.5:0/724293257 wait complete. 2024-03-29T10:49:45.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:45.303+0000 7f185effd640 1 librados: shutdown 2024-03-29T10:49:45.305 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:52.112 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:52.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.283+0000 7f490cd41640 10 monclient: get_monmap_and_config 2024-03-29T10:49:52.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.283+0000 7f490cd41640 10 monclient: build_initial_monmap 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.283+0000 7f490cd41640 10 monclient: monmap: 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:49:52.284548+0000 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:49:52.284548+0000 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:49:52.267 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:52.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.286+0000 7f490cd41640 1 Processor -- start 2024-03-29T10:49:52.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.286+0000 7f490cd41640 1 -- start start 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.286+0000 7f490cd41640 10 monclient: init 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.287+0000 7f490cd41640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.287+0000 7f490cd41640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.287+0000 7f490cd41640 10 monclient(hunting): picked mon.noname-c con 0x7f4900034850 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.287+0000 7f490cd41640 10 monclient(hunting): picked mon.noname-a con 0x7f49000c9550 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.287+0000 7f490cd41640 10 monclient(hunting): picked mon.noname-b con 0x7f490002aac0 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.287+0000 7f490cd41640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490002c440 con 0x7f49000c9550 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.287+0000 7f490cd41640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490002aed0 con 0x7f4900034850 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.287+0000 7f490cd41640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490002b190 con 0x7f490002aac0 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.287+0000 7f490cd41640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:52.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.287+0000 7f490cd41640 10 monclient(hunting): authenticate will time out at 1005.528266s 2024-03-29T10:49:52.275 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.293+0000 7f4906ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f49000c9550 0x7f49000cbea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38041/0 (socket says 172.21.15.5:38041) 2024-03-29T10:49:52.275 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.293+0000 7f4906ffd640 1 -- 172.21.15.5:0/2587409119 learned_addr learned my addr 172.21.15.5:0/2587409119 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:52.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.294+0000 7f49067fc640 1 -- 172.21.15.5:0/2587409119 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231717090 0 0) 0x7f490002c440 con 0x7f49000c9550 2024-03-29T10:49:52.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.294+0000 7f49067fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:52.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.294+0000 7f49067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:52.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.294+0000 7f49067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:52.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.294+0000 7f49067fc640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:49:52.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.294+0000 7f49067fc640 1 -- 172.21.15.5:0/2587409119 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f8003520 con 0x7f49000c9550 2024-03-29T10:49:52.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 1 -- 172.21.15.5:0/2587409119 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 245298426 0 0) 0x7f48f8003520 con 0x7f49000c9550 2024-03-29T10:49:52.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 1 -- 172.21.15.5:0/2587409119 >> v1:172.21.15.5:6790/0 conn(0x7f490002aac0 legacy=0x7f49000cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:52.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 1 -- 172.21.15.5:0/2587409119 >> v1:172.21.15.177:6789/0 conn(0x7f4900034850 legacy=0x7f49000c8e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:52.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 1 monclient: found mon.noname-a 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 1 -- 172.21.15.5:0/2587409119 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4900033060 con 0x7f49000c9550 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 1 -- 172.21.15.5:0/2587409119 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f48f4001690 con 0x7f49000c9550 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f490cd41640 5 monclient: authenticate success, global_id 4290 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f490cd41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: dump: 2024-03-29T10:49:52.282 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f490cd41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 1 -- 172.21.15.5:0/2587409119 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48f4001860 con 0x7f49000c9550 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 1 -- 172.21.15.5:0/2587409119 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f48f4001cb0 con 0x7f49000c9550 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f49067fc640 10 monclient: dump: 2024-03-29T10:49:52.283 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f490cd41640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.295+0000 7f490cd41640 4 set_mon_vals no callback set 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.296+0000 7f490cd41640 10 monclient: shutdown 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.296+0000 7f490cd41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:52.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.296+0000 7f490cd41640 1 -- 172.21.15.5:0/2587409119 >> v1:172.21.15.5:6789/0 conn(0x7f49000c9550 legacy=0x7f49000cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.296+0000 7f490cd41640 1 -- 172.21.15.5:0/2587409119 shutdown_connections 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.296+0000 7f490cd41640 1 -- 172.21.15.5:0/2587409119 >> 172.21.15.5:0/2587409119 conn(0x7f49000058f0 msgr2=0x7f4900005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.297+0000 7f490cd41640 1 -- 172.21.15.5:0/2587409119 shutdown_connections 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.297+0000 7f490cd41640 1 -- 172.21.15.5:0/2587409119 wait complete. 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.298+0000 7f490cd41640 10 monclient: build_initial_monmap 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.298+0000 7f490cd41640 10 monclient: monmap: 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:49:52.285 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:52.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.298+0000 7f490cd41640 1 librados: starting msgr at 2024-03-29T10:49:52.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.298+0000 7f490cd41640 1 librados: starting objecter 2024-03-29T10:49:52.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.298+0000 7f490cd41640 1 Processor -- start 2024-03-29T10:49:52.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.299+0000 7f490cd41640 1 -- start start 2024-03-29T10:49:52.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.299+0000 7f490cd41640 1 librados: setting wanted keys 2024-03-29T10:49:52.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.299+0000 7f490cd41640 1 librados: calling monclient init 2024-03-29T10:49:52.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.299+0000 7f490cd41640 10 monclient: init 2024-03-29T10:49:52.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.300+0000 7f490cd41640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.300+0000 7f490cd41640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.300+0000 7f490cd41640 10 monclient(hunting): picked mon.noname-a con 0x7f490002aac0 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.300+0000 7f490cd41640 10 monclient(hunting): picked mon.noname-b con 0x7f4900034850 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.300+0000 7f490cd41640 10 monclient(hunting): picked mon.noname-c con 0x7f49000c9550 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.300+0000 7f490cd41640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49000a1460 con 0x7f490002aac0 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.300+0000 7f490cd41640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49000a1740 con 0x7f4900034850 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.300+0000 7f490cd41640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49000329e0 con 0x7f49000c9550 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.300+0000 7f490cd41640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.300+0000 7f490cd41640 10 monclient(hunting): authenticate will time out at 1005.541775s 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.305+0000 7f4907fff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f49000c9550 0x7f4900175070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:41249/0 (socket says 172.21.15.5:41249) 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.305+0000 7f4907fff640 1 -- 172.21.15.5:0/1981433589 learned_addr learned my addr 172.21.15.5:0/1981433589 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.305+0000 7f48f3fff640 10 client.?.objecter ms_handle_connect 0x7f49000c9550 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.305+0000 7f48f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:52.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.305+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506860090 0 0) 0x7f49000329e0 con 0x7f49000c9550 2024-03-29T10:49:52.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.305+0000 7f48f3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:52.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.305+0000 7f48f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:52.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.305+0000 7f48f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:52.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.305+0000 7f48f3fff640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:49:52.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.305+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ec004eb0 con 0x7f49000c9550 2024-03-29T10:49:52.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.309+0000 7f48f3fff640 10 client.?.objecter ms_handle_connect 0x7f490002aac0 2024-03-29T10:49:52.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.309+0000 7f48f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:52.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.309+0000 7f48f3fff640 10 client.?.objecter ms_handle_connect 0x7f4900034850 2024-03-29T10:49:52.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.309+0000 7f48f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:52.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.309+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 290079026 0 0) 0x7f48ec004eb0 con 0x7f49000c9550 2024-03-29T10:49:52.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49000329e0 con 0x7f49000c9550 2024-03-29T10:49:52.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f48fc0018e0 con 0x7f49000c9550 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): dump: 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:52.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536806852 0 0) 0x7f49000a1740 con 0x7f4900034850 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ec004eb0 con 0x7f4900034850 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742301310 0 0) 0x7f49000a1460 con 0x7f490002aac0 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49000a1740 con 0x7f490002aac0 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3744872906 0 0) 0x7f49000a1740 con 0x7f490002aac0 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49000a1460 con 0x7f490002aac0 2024-03-29T10:49:52.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f490804fdd0 con 0x7f490002aac0 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.310+0000 7f48f3fff640 10 monclient(hunting): dump: 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:52.295 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:52.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2207984039 0 0) 0x7f49000a1460 con 0x7f490002aac0 2024-03-29T10:49:52.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 >> v1:172.21.15.5:6790/0 conn(0x7f49000c9550 legacy=0x7f4900175070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:52.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 >> v1:172.21.15.177:6789/0 conn(0x7f4900034850 legacy=0x7f4900169bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:52.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:52.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 1 monclient: found mon.a 2024-03-29T10:49:52.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:52.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:52.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49000a0890 con 0x7f490002aac0 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 10 monclient: _finish_auth 0 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f48f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f490cd41640 5 monclient: authenticate success, global_id 4293 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f490cd41640 10 monclient: _renew_subs 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f490cd41640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f490cd41640 1 -- 172.21.15.5:0/1981433589 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4900032030 con 0x7f490002aac0 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f490cd41640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f490cd41640 10 monclient: _renew_subs 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f490cd41640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f490cd41640 1 -- 172.21.15.5:0/1981433589 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f490002f910 con 0x7f490002aac0 2024-03-29T10:49:52.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.311+0000 7f490cd41640 1 librados: init done 2024-03-29T10:49:52.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.314+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f490804f400 con 0x7f490002aac0 2024-03-29T10:49:52.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.314+0000 7f48f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:52.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.314+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4908075710 con 0x7f490002aac0 2024-03-29T10:49:52.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.314+0000 7f48f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:52.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.314+0000 7f48f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:52.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.314+0000 7f48f3fff640 10 monclient: dump: 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.314+0000 7f49057fa640 4 set_mon_vals no callback set 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.314+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f4908067bf0 con 0x7f490002aac0 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.315+0000 7f490cd41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.315+0000 7f490cd41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:52.300 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.315+0000 7f490cd41640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:52.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.315+0000 7f490cd41640 1 -- 172.21.15.5:0/1981433589 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e8005080 con 0x7f490002aac0 2024-03-29T10:49:52.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.315+0000 7f4906ffd640 10 monclient: get_auth_request con 0x7f48ec06df10 auth_method 0 2024-03-29T10:49:52.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.315+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f49080fe0e0 con 0x7f490002aac0 2024-03-29T10:49:52.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.315+0000 7f48f3fff640 10 client.4293.objecter ms_dispatch 0x7f4908000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:52.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.315+0000 7f48f3fff640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:52.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.315+0000 7f48f3fff640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:52.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.315+0000 7f48f3fff640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-29T10:49:52.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.316+0000 7f48f3fff640 10 client.4293.objecter ms_handle_connect 0x7f48ec06df10 2024-03-29T10:49:52.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.325+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49080c9810 con 0x7f490002aac0 2024-03-29T10:49:52.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.325+0000 7f48f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:52.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.325+0000 7f48f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:52.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.654+0000 7f490cd41640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:52.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.654+0000 7f490cd41640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:52.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.654+0000 7f490cd41640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:52.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.654+0000 7f490cd41640 1 -- 172.21.15.5:0/1981433589 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48e8005330 con 0x7f490002aac0 2024-03-29T10:49:52.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.654+0000 7f48f3fff640 1 -- 172.21.15.5:0/1981433589 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f49080cb8f0 con 0x7f490002aac0 2024-03-29T10:49:52.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.655+0000 7f48f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:52.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.655+0000 7f48f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:52.639 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:49:52.640 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:52.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.659+0000 7f48f1ffb640 10 librados: watch_flush enter 2024-03-29T10:49:52.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.659+0000 7f48f1ffb640 10 librados: watch_flush exit 2024-03-29T10:49:52.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.659+0000 7f48f1ffb640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-29T10:49:52.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.659+0000 7f48f1ffb640 10 client.4293.objecter successfully canceled tick 2024-03-29T10:49:52.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.659+0000 7f48f1ffb640 1 -- 172.21.15.5:0/1981433589 >> v1:172.21.15.5:6816/2847500151 conn(0x7f48ec06df10 legacy=0x7f48ec0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:52.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.659+0000 7f48f1ffb640 10 monclient: shutdown 2024-03-29T10:49:52.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.659+0000 7f48f1ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:52.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.659+0000 7f48f1ffb640 1 -- 172.21.15.5:0/1981433589 >> v1:172.21.15.5:6789/0 conn(0x7f490002aac0 legacy=0x7f490002f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:52.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.659+0000 7f48f1ffb640 1 -- 172.21.15.5:0/1981433589 shutdown_connections 2024-03-29T10:49:52.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.659+0000 7f48f1ffb640 1 -- 172.21.15.5:0/1981433589 >> 172.21.15.5:0/1981433589 conn(0x7f49000058f0 msgr2=0x7f49000043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:52.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.660+0000 7f48f1ffb640 1 -- 172.21.15.5:0/1981433589 shutdown_connections 2024-03-29T10:49:52.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.660+0000 7f48f1ffb640 1 -- 172.21.15.5:0/1981433589 wait complete. 2024-03-29T10:49:52.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:52.660+0000 7f48f1ffb640 1 librados: shutdown 2024-03-29T10:49:52.659 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:59.466 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:59.620 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.638+0000 7f68eada0640 10 monclient: get_monmap_and_config 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.638+0000 7f68eada0640 10 monclient: build_initial_monmap 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.638+0000 7f68eada0640 10 monclient: monmap: 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:49:59.639787+0000 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:49:59.639787+0000 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:49:59.621 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:59.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.639+0000 7f68eada0640 1 Processor -- start 2024-03-29T10:49:59.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.640+0000 7f68eada0640 1 -- start start 2024-03-29T10:49:59.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.640+0000 7f68eada0640 10 monclient: init 2024-03-29T10:49:59.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.640+0000 7f68eada0640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:59.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.640+0000 7f68eada0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:59.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.644+0000 7f68eada0640 10 monclient(hunting): picked mon.noname-c con 0x7f68dc006a30 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:59.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.644+0000 7f68eada0640 10 monclient(hunting): picked mon.noname-b con 0x7f68dc0360a0 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:59.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.644+0000 7f68e9d9e640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f68dc006a30 0x7f68dc0cf1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:46223/0 (socket says 172.21.15.5:46223) 2024-03-29T10:49:59.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.646+0000 7f68eada0640 10 monclient(hunting): picked mon.noname-a con 0x7f68dc0c9690 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:59.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.646+0000 7f68eada0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68dc034d40 con 0x7f68dc0c9690 2024-03-29T10:49:59.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.646+0000 7f68eada0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68dc035020 con 0x7f68dc006a30 2024-03-29T10:49:59.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.646+0000 7f68eada0640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68dc034300 con 0x7f68dc0360a0 2024-03-29T10:49:59.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.646+0000 7f68eada0640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:59.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.646+0000 7f68eada0640 10 monclient(hunting): authenticate will time out at 1012.887807s 2024-03-29T10:49:59.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.646+0000 7f68e9d9e640 1 -- 172.21.15.5:0/2785221767 learned_addr learned my addr 172.21.15.5:0/2785221767 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:59.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.647+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269495469 0 0) 0x7f68dc035020 con 0x7f68dc006a30 2024-03-29T10:49:59.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.647+0000 7f68e8d9c640 10 monclient(hunting): _negotiate 2024-03-29T10:49:59.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.647+0000 7f68e8d9c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:59.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.647+0000 7f68e8d9c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:59.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.647+0000 7f68e8d9c640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:49:59.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.647+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d0003520 con 0x7f68dc006a30 2024-03-29T10:49:59.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.648+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353290241 0 0) 0x7f68dc034300 con 0x7f68dc0360a0 2024-03-29T10:49:59.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.648+0000 7f68e8d9c640 10 monclient(hunting): _negotiate 2024-03-29T10:49:59.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.648+0000 7f68e8d9c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:59.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.648+0000 7f68e8d9c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:59.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.648+0000 7f68e8d9c640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:49:59.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.648+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68dc035020 con 0x7f68dc0360a0 2024-03-29T10:49:59.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.649+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4180388732 0 0) 0x7f68d0003520 con 0x7f68dc006a30 2024-03-29T10:49:59.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.649+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 >> v1:172.21.15.5:6790/0 conn(0x7f68dc0360a0 legacy=0x7f68dc0c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:59.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.649+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 >> v1:172.21.15.5:6789/0 conn(0x7f68dc0c9690 legacy=0x7f68dc0d7540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:59.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: _finish_hunting 0 2024-03-29T10:49:59.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 1 monclient: found mon.noname-c 2024-03-29T10:49:59.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:59.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:49:59.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc034490 con 0x7f68dc006a30 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: _finish_auth 0 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: _check_auth_tickets 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f68e0001690 con 0x7f68dc006a30 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: dump: 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:59.634 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68e00018b0 con 0x7f68dc006a30 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 1 -- 172.21.15.5:0/2785221767 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f68e0001c40 con 0x7f68dc006a30 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.650+0000 7f68e8d9c640 10 monclient: dump: 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:59.635 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:59.636 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:59.636 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:59.636 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:59.636 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:59.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.653+0000 7f68eada0640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:49:59.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.653+0000 7f68eada0640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:59.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.653+0000 7f68eada0640 4 set_mon_vals no callback set 2024-03-29T10:49:59.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.654+0000 7f68eada0640 10 monclient: shutdown 2024-03-29T10:49:59.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.654+0000 7f68eada0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:59.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.654+0000 7f68eada0640 1 -- 172.21.15.5:0/2785221767 >> v1:172.21.15.177:6789/0 conn(0x7f68dc006a30 legacy=0x7f68dc0cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:59.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.654+0000 7f68eada0640 1 -- 172.21.15.5:0/2785221767 shutdown_connections 2024-03-29T10:49:59.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.654+0000 7f68eada0640 1 -- 172.21.15.5:0/2785221767 >> 172.21.15.5:0/2785221767 conn(0x7f68dc005ca0 msgr2=0x7f68dc0060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:59.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.657+0000 7f68eada0640 1 -- 172.21.15.5:0/2785221767 shutdown_connections 2024-03-29T10:49:59.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.658+0000 7f68eada0640 1 -- 172.21.15.5:0/2785221767 wait complete. 2024-03-29T10:49:59.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.659+0000 7f68eada0640 10 monclient: build_initial_monmap 2024-03-29T10:49:59.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.659+0000 7f68eada0640 10 monclient: monmap: 2024-03-29T10:49:59.641 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:49:59.641 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:59.641 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:49:59.641 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:49:59.642 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:59.642 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:59.642 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:49:59.642 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:49:59.642 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:49:59.642 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:59.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.659+0000 7f68eada0640 1 librados: starting msgr at 2024-03-29T10:49:59.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.659+0000 7f68eada0640 1 librados: starting objecter 2024-03-29T10:49:59.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.659+0000 7f68eada0640 1 Processor -- start 2024-03-29T10:49:59.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.659+0000 7f68eada0640 1 -- start start 2024-03-29T10:49:59.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.660+0000 7f68eada0640 1 librados: setting wanted keys 2024-03-29T10:49:59.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.660+0000 7f68eada0640 1 librados: calling monclient init 2024-03-29T10:49:59.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.660+0000 7f68eada0640 10 monclient: init 2024-03-29T10:49:59.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.660+0000 7f68eada0640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:59.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.660+0000 7f68eada0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:59.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.662+0000 7f68eada0640 10 monclient(hunting): picked mon.noname-c con 0x7f68dc006a30 addr v1:172.21.15.5:6790/0 2024-03-29T10:49:59.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.662+0000 7f68eada0640 10 monclient(hunting): picked mon.noname-b con 0x7f68dc0360a0 addr v1:172.21.15.177:6789/0 2024-03-29T10:49:59.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.662+0000 7f68eada0640 10 monclient(hunting): picked mon.noname-a con 0x7f68dc0c9690 addr v1:172.21.15.5:6789/0 2024-03-29T10:49:59.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.662+0000 7f68eada0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68dc0c4560 con 0x7f68dc0c9690 2024-03-29T10:49:59.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.662+0000 7f68eada0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68dc164f60 con 0x7f68dc0360a0 2024-03-29T10:49:59.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.662+0000 7f68eada0640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68dc0cbbe0 con 0x7f68dc006a30 2024-03-29T10:49:59.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.662+0000 7f68eada0640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:59.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.662+0000 7f68eada0640 10 monclient(hunting): authenticate will time out at 1012.903315s 2024-03-29T10:49:59.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.667+0000 7f68e9d9e640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f68dc006a30 0x7f68dc00afe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:39337/0 (socket says 172.21.15.5:39337) 2024-03-29T10:49:59.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.667+0000 7f68e9d9e640 1 -- 172.21.15.5:0/2801553334 learned_addr learned my addr 172.21.15.5:0/2801553334 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.667+0000 7f68da7fc640 10 client.?.objecter ms_handle_connect 0x7f68dc006a30 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.667+0000 7f68da7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.667+0000 7f68da7fc640 10 client.?.objecter ms_handle_connect 0x7f68dc0c9690 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.667+0000 7f68da7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840701292 0 0) 0x7f68dc0cbbe0 con 0x7f68dc006a30 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c8004eb0 con 0x7f68dc006a30 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628279926 0 0) 0x7f68dc0c4560 con 0x7f68dc0c9690 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:59.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:59.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:59.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:49:59.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.668+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68dc0cbbe0 con 0x7f68dc0c9690 2024-03-29T10:49:59.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.670+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 161500225 0 0) 0x7f68c8004eb0 con 0x7f68dc006a30 2024-03-29T10:49:59.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68dc0c4560 con 0x7f68dc006a30 2024-03-29T10:49:59.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 10 client.?.objecter ms_handle_connect 0x7f68dc0360a0 2024-03-29T10:49:59.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 691687800 0 0) 0x7f68dc0cbbe0 con 0x7f68dc0c9690 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68c8004eb0 con 0x7f68dc0c9690 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f68e0001690 con 0x7f68dc006a30 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 10 monclient(hunting): dump: 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:59.655 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f68e404fdd0 con 0x7f68dc0c9690 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.671+0000 7f68da7fc640 10 monclient(hunting): dump: 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:59.656 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2291596962 0 0) 0x7f68c8004eb0 con 0x7f68dc0c9690 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 >> v1:172.21.15.5:6790/0 conn(0x7f68dc006a30 legacy=0x7f68dc00afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 >> v1:172.21.15.177:6789/0 conn(0x7f68dc0360a0 legacy=0x7f68dc0c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 1 monclient: found mon.a 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc0cbec0 con 0x7f68dc0c9690 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.672+0000 7f68da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:59.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68e404f400 con 0x7f68dc0c9690 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f68e4075f10 con 0x7f68dc0c9690 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68da7fc640 10 monclient: dump: 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:49:59.658 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:49:59.659 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:49:59.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68eada0640 5 monclient: authenticate success, global_id 4296 2024-03-29T10:49:59.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68eada0640 10 monclient: _renew_subs 2024-03-29T10:49:59.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68eada0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:59.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68eada0640 1 -- 172.21.15.5:0/2801553334 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68dc0cac10 con 0x7f68dc0c9690 2024-03-29T10:49:59.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68eada0640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:59.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68eada0640 10 monclient: _renew_subs 2024-03-29T10:49:59.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68eada0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68eada0640 1 -- 172.21.15.5:0/2801553334 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68dc00d8e0 con 0x7f68dc0c9690 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.673+0000 7f68eada0640 1 librados: init done 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.675+0000 7f68dbfff640 4 set_mon_vals no callback set 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.675+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f68e4067bf0 con 0x7f68dc0c9690 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.676+0000 7f68e9d9e640 10 monclient: get_auth_request con 0x7f68c806ddc0 auth_method 0 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.676+0000 7f68eada0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.676+0000 7f68eada0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.676+0000 7f68eada0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.676+0000 7f68eada0640 1 -- 172.21.15.5:0/2801553334 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68cc005080 con 0x7f68dc0c9690 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.677+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f68e40fdf50 con 0x7f68dc0c9690 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.677+0000 7f68da7fc640 10 client.4296.objecter ms_dispatch 0x7f68e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.677+0000 7f68da7fc640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.677+0000 7f68da7fc640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:59.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.677+0000 7f68da7fc640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T10:49:59.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.677+0000 7f68da7fc640 10 client.4296.objecter ms_handle_connect 0x7f68c806ddc0 2024-03-29T10:49:59.669 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.687+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f68e40c9930 con 0x7f68dc0c9690 2024-03-29T10:49:59.669 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.687+0000 7f68da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:59.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:49:59.687+0000 7f68da7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:59.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.015+0000 7f68eada0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:59.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.015+0000 7f68eada0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:59.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.015+0000 7f68eada0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:49:59.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.015+0000 7f68eada0640 1 -- 172.21.15.5:0/2801553334 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68cc005330 con 0x7f68dc0c9690 2024-03-29T10:50:00.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.019+0000 7f68da7fc640 1 -- 172.21.15.5:0/2801553334 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f68e40cb860 con 0x7f68dc0c9690 2024-03-29T10:50:00.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.019+0000 7f68da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:00.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.019+0000 7f68da7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:00.002 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:50:00.002 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:00.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 10 librados: watch_flush enter 2024-03-29T10:50:00.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 10 librados: watch_flush exit 2024-03-29T10:50:00.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T10:50:00.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 10 client.4296.objecter successfully canceled tick 2024-03-29T10:50:00.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 1 -- 172.21.15.5:0/2801553334 >> v1:172.21.15.5:6816/2847500151 conn(0x7f68c806ddc0 legacy=0x7f68c8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:00.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 10 monclient: shutdown 2024-03-29T10:50:00.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:00.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 1 -- 172.21.15.5:0/2801553334 >> v1:172.21.15.5:6789/0 conn(0x7f68dc0c9690 legacy=0x7f68dc0c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:00.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 1 -- 172.21.15.5:0/2801553334 shutdown_connections 2024-03-29T10:50:00.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 1 -- 172.21.15.5:0/2801553334 >> 172.21.15.5:0/2801553334 conn(0x7f68dc005ca0 msgr2=0x7f68dc0c6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:00.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 1 -- 172.21.15.5:0/2801553334 shutdown_connections 2024-03-29T10:50:00.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.023+0000 7f68eada0640 1 -- 172.21.15.5:0/2801553334 wait complete. 2024-03-29T10:50:00.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:00.024+0000 7f68eada0640 1 librados: shutdown 2024-03-29T10:50:00.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:06.830 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:06.989 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.006+0000 7f35eb174640 10 monclient: get_monmap_and_config 2024-03-29T10:50:06.989 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.007+0000 7f35eb174640 10 monclient: build_initial_monmap 2024-03-29T10:50:06.989 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.007+0000 7f35eb174640 10 monclient: monmap: 2024-03-29T10:50:06.989 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:06.990 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:06.990 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:50:07.008044+0000 2024-03-29T10:50:06.990 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:50:07.008044+0000 2024-03-29T10:50:06.990 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:06.990 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:06.990 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:06.990 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:50:06.990 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:50:06.990 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:06.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.008+0000 7f35eb174640 1 Processor -- start 2024-03-29T10:50:06.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.008+0000 7f35eb174640 1 -- start start 2024-03-29T10:50:06.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.008+0000 7f35eb174640 10 monclient: init 2024-03-29T10:50:06.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.009+0000 7f35eb174640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:06.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.009+0000 7f35eb174640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:06.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.009+0000 7f35eb174640 10 monclient(hunting): picked mon.noname-b con 0x7f35dc006800 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:06.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.009+0000 7f35eb174640 10 monclient(hunting): picked mon.noname-a con 0x7f35dc006be0 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:06.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.009+0000 7f35eb174640 10 monclient(hunting): picked mon.noname-c con 0x7f35dc031b60 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:06.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.009+0000 7f35eb174640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc029370 con 0x7f35dc006be0 2024-03-29T10:50:06.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.009+0000 7f35eb174640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc0295e0 con 0x7f35dc031b60 2024-03-29T10:50:06.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.009+0000 7f35eb174640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc031360 con 0x7f35dc006800 2024-03-29T10:50:06.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.009+0000 7f35eb174640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:06.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.009+0000 7f35eb174640 10 monclient(hunting): authenticate will time out at 1020.250335s 2024-03-29T10:50:06.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.010+0000 7f35ea172640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f35dc006800 0x7f35dc0cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:36138/0 (socket says 172.21.15.5:36138) 2024-03-29T10:50:06.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.010+0000 7f35ea172640 1 -- 172.21.15.5:0/3150719350 learned_addr learned my addr 172.21.15.5:0/3150719350 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:06.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589610790 0 0) 0x7f35dc031360 con 0x7f35dc006800 2024-03-29T10:50:06.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 10 monclient(hunting): _negotiate 2024-03-29T10:50:06.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:06.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:06.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:50:06.996 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d0003520 con 0x7f35dc006800 2024-03-29T10:50:06.996 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313429309 0 0) 0x7f35dc029370 con 0x7f35dc006be0 2024-03-29T10:50:06.996 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 10 monclient(hunting): _negotiate 2024-03-29T10:50:06.996 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:06.996 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:06.999 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:50:06.999 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.011+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35dc031360 con 0x7f35dc006be0 2024-03-29T10:50:06.999 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.012+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3823837931 0 0) 0x7f35d0003520 con 0x7f35dc006800 2024-03-29T10:50:06.999 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.012+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 >> v1:172.21.15.177:6789/0 conn(0x7f35dc031b60 legacy=0x7f35dc0d30c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:06.999 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 >> v1:172.21.15.5:6789/0 conn(0x7f35dc006be0 legacy=0x7f35dc0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:06.999 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 10 monclient: _finish_hunting 0 2024-03-29T10:50:06.999 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 1 monclient: found mon.noname-b 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc031640 con 0x7f35dc006800 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 10 monclient: _finish_auth 0 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 10 monclient: _check_auth_tickets 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f35e0001690 con 0x7f35dc006800 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35e9170640 10 monclient: dump: 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:07.000 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35eb174640 5 monclient: authenticate success, global_id 4328 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.013+0000 7f35eb174640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.015+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e00017f0 con 0x7f35dc006800 2024-03-29T10:50:07.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.015+0000 7f35e9170640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.015+0000 7f35e9170640 1 -- 172.21.15.5:0/3150719350 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f35e0001bc0 con 0x7f35dc006800 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.015+0000 7f35eb174640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.015+0000 7f35eb174640 4 set_mon_vals no callback set 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.015+0000 7f35e9170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.015+0000 7f35e9170640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.015+0000 7f35e9170640 10 monclient: dump: 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:07.002 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:07.003 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:07.003 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:07.003 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:07.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.016+0000 7f35eb174640 10 monclient: shutdown 2024-03-29T10:50:07.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.016+0000 7f35eb174640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:07.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.016+0000 7f35eb174640 1 -- 172.21.15.5:0/3150719350 >> v1:172.21.15.5:6790/0 conn(0x7f35dc006800 legacy=0x7f35dc0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:07.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.016+0000 7f35eb174640 1 -- 172.21.15.5:0/3150719350 shutdown_connections 2024-03-29T10:50:07.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.016+0000 7f35eb174640 1 -- 172.21.15.5:0/3150719350 >> 172.21.15.5:0/3150719350 conn(0x7f35dc0058f0 msgr2=0x7f35dc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:07.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.016+0000 7f35eb174640 1 -- 172.21.15.5:0/3150719350 shutdown_connections 2024-03-29T10:50:07.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.016+0000 7f35eb174640 1 -- 172.21.15.5:0/3150719350 wait complete. 2024-03-29T10:50:07.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.017+0000 7f35eb174640 10 monclient: build_initial_monmap 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.017+0000 7f35eb174640 10 monclient: monmap: 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:50:07.010 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.018+0000 7f35eb174640 1 librados: starting msgr at 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.018+0000 7f35eb174640 1 librados: starting objecter 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.018+0000 7f35eb174640 1 Processor -- start 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.018+0000 7f35eb174640 1 -- start start 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.018+0000 7f35eb174640 1 librados: setting wanted keys 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.018+0000 7f35eb174640 1 librados: calling monclient init 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.018+0000 7f35eb174640 10 monclient: init 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.019+0000 7f35eb174640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.019+0000 7f35eb174640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.019+0000 7f35eb174640 10 monclient(hunting): picked mon.noname-b con 0x7f35dc006800 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.019+0000 7f35eb174640 10 monclient(hunting): picked mon.noname-c con 0x7f35dc031b60 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:07.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.019+0000 7f35eb174640 10 monclient(hunting): picked mon.noname-a con 0x7f35dc02e6b0 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.019+0000 7f35eb174640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc16c900 con 0x7f35dc02e6b0 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.019+0000 7f35eb174640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc0c7360 con 0x7f35dc006800 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.019+0000 7f35eb174640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc0c55a0 con 0x7f35dc031b60 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.019+0000 7f35eb174640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.019+0000 7f35eb174640 10 monclient(hunting): authenticate will time out at 1020.260999s 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.020+0000 7f35ea973640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f35dc02e6b0 0x7f35dc1663c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50542/0 (socket says 172.21.15.5:50542) 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.020+0000 7f35ea973640 1 -- 172.21.15.5:0/100550427 learned_addr learned my addr 172.21.15.5:0/100550427 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.020+0000 7f35daffd640 10 client.?.objecter ms_handle_connect 0x7f35dc02e6b0 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.020+0000 7f35daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.020+0000 7f35daffd640 10 client.?.objecter ms_handle_connect 0x7f35dc031b60 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.020+0000 7f35daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:07.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029217907 0 0) 0x7f35dc0c55a0 con 0x7f35dc031b60 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c8004eb0 con 0x7f35dc031b60 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365708282 0 0) 0x7f35dc16c900 con 0x7f35dc02e6b0 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35dc0c55a0 con 0x7f35dc02e6b0 2024-03-29T10:50:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2022421125 0 0) 0x7f35dc0c55a0 con 0x7f35dc02e6b0 2024-03-29T10:50:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35dc16c900 con 0x7f35dc02e6b0 2024-03-29T10:50:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f35e404fdd0 con 0x7f35dc02e6b0 2024-03-29T10:50:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): dump: 2024-03-29T10:50:07.014 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:07.014 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:07.014 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:07.014 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:07.014 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987441392 0 0) 0x7f35c8004eb0 con 0x7f35dc031b60 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35dc0c55a0 con 0x7f35dc031b60 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f35d4001a10 con 0x7f35dc031b60 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.021+0000 7f35daffd640 10 monclient(hunting): dump: 2024-03-29T10:50:07.015 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.023+0000 7f35daffd640 10 client.?.objecter ms_handle_connect 0x7f35dc006800 2024-03-29T10:50:07.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.023+0000 7f35daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.023+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2765415841 0 0) 0x7f35dc16c900 con 0x7f35dc02e6b0 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.023+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 >> v1:172.21.15.5:6790/0 conn(0x7f35dc031b60 legacy=0x7f35dc16cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.023+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 >> v1:172.21.15.177:6789/0 conn(0x7f35dc006800 legacy=0x7f35dc0c6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.023+0000 7f35daffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.023+0000 7f35daffd640 1 monclient: found mon.a 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.023+0000 7f35daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.023+0000 7f35daffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.023+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc0c75d0 con 0x7f35dc02e6b0 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35daffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35daffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35eb174640 5 monclient: authenticate success, global_id 4302 2024-03-29T10:50:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35eb174640 10 monclient: _renew_subs 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35eb174640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35eb174640 1 -- 172.21.15.5:0/100550427 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35dc16d4a0 con 0x7f35dc02e6b0 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35eb174640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35eb174640 10 monclient: _renew_subs 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35eb174640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35eb174640 1 -- 172.21.15.5:0/100550427 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35dc166cb0 con 0x7f35dc02e6b0 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35eb174640 1 librados: init done 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e404f400 con 0x7f35dc02e6b0 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f35e4075f10 con 0x7f35dc02e6b0 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:07.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35daffd640 10 monclient: dump: 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:07.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.024+0000 7f35e896f640 4 set_mon_vals no callback set 2024-03-29T10:50:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.027+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f35e4067bf0 con 0x7f35dc02e6b0 2024-03-29T10:50:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.027+0000 7f35eb174640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.027+0000 7f35eb174640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.027+0000 7f35eb174640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.027+0000 7f35eb174640 1 -- 172.21.15.5:0/100550427 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35cc005080 con 0x7f35dc02e6b0 2024-03-29T10:50:07.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.028+0000 7f35ea172640 10 monclient: get_auth_request con 0x7f35c806dd50 auth_method 0 2024-03-29T10:50:07.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.029+0000 7f35daffd640 10 client.4302.objecter ms_handle_connect 0x7f35c806dd50 2024-03-29T10:50:07.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.029+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f35e40ff3a0 con 0x7f35dc02e6b0 2024-03-29T10:50:07.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.029+0000 7f35daffd640 10 client.4302.objecter ms_dispatch 0x7f35e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:07.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.029+0000 7f35daffd640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:07.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.029+0000 7f35daffd640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:07.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.029+0000 7f35daffd640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T10:50:07.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.038+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35e40cac80 con 0x7f35dc02e6b0 2024-03-29T10:50:07.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.039+0000 7f35daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:07.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.039+0000 7f35daffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:07.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.408+0000 7f35eb174640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:07.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.408+0000 7f35eb174640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:07.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.408+0000 7f35eb174640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:07.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.408+0000 7f35eb174640 1 -- 172.21.15.5:0/100550427 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35cc005330 con 0x7f35dc02e6b0 2024-03-29T10:50:07.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.410+0000 7f35daffd640 1 -- 172.21.15.5:0/100550427 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f35e40ccbb0 con 0x7f35dc02e6b0 2024-03-29T10:50:07.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.410+0000 7f35daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:07.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.410+0000 7f35daffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:07.392 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:50:07.392 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:07.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.414+0000 7f35eb174640 10 librados: watch_flush enter 2024-03-29T10:50:07.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.414+0000 7f35eb174640 10 librados: watch_flush exit 2024-03-29T10:50:07.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.414+0000 7f35eb174640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T10:50:07.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.414+0000 7f35eb174640 10 client.4302.objecter successfully canceled tick 2024-03-29T10:50:07.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.415+0000 7f35eb174640 1 -- 172.21.15.5:0/100550427 >> v1:172.21.15.5:6816/2847500151 conn(0x7f35c806dd50 legacy=0x7f35c8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:07.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.415+0000 7f35eb174640 10 monclient: shutdown 2024-03-29T10:50:07.398 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.415+0000 7f35eb174640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:07.398 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.415+0000 7f35eb174640 1 -- 172.21.15.5:0/100550427 >> v1:172.21.15.5:6789/0 conn(0x7f35dc02e6b0 legacy=0x7f35dc1663c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:07.398 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.415+0000 7f35eb174640 1 -- 172.21.15.5:0/100550427 shutdown_connections 2024-03-29T10:50:07.398 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.415+0000 7f35eb174640 1 -- 172.21.15.5:0/100550427 >> 172.21.15.5:0/100550427 conn(0x7f35dc0058f0 msgr2=0x7f35dc005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:07.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.415+0000 7f35eb174640 1 -- 172.21.15.5:0/100550427 shutdown_connections 2024-03-29T10:50:07.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.416+0000 7f35eb174640 1 -- 172.21.15.5:0/100550427 wait complete. 2024-03-29T10:50:07.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:07.416+0000 7f35eb174640 1 librados: shutdown 2024-03-29T10:50:07.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:14.221 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:14.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.399+0000 7f18db510640 10 monclient: get_monmap_and_config 2024-03-29T10:50:14.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.399+0000 7f18db510640 10 monclient: build_initial_monmap 2024-03-29T10:50:14.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.399+0000 7f18db510640 10 monclient: monmap: 2024-03-29T10:50:14.381 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:14.381 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:14.381 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:50:14.400362+0000 2024-03-29T10:50:14.381 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:50:14.400362+0000 2024-03-29T10:50:14.382 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:14.382 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:14.382 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:14.382 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:50:14.382 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:50:14.382 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:14.382 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.400+0000 7f18db510640 1 Processor -- start 2024-03-29T10:50:14.382 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.400+0000 7f18db510640 1 -- start start 2024-03-29T10:50:14.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.400+0000 7f18db510640 10 monclient: init 2024-03-29T10:50:14.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.401+0000 7f18db510640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:14.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.401+0000 7f18db510640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:14.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.401+0000 7f18db510640 10 monclient(hunting): picked mon.noname-a con 0x7f18d40b1f60 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:14.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.401+0000 7f18db510640 10 monclient(hunting): picked mon.noname-c con 0x7f18d4071eb0 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:14.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.401+0000 7f18db510640 10 monclient(hunting): picked mon.noname-b con 0x7f18d4143d10 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:14.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.401+0000 7f18db510640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d406ef80 con 0x7f18d40b1f60 2024-03-29T10:50:14.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.401+0000 7f18db510640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d406f260 con 0x7f18d4071eb0 2024-03-29T10:50:14.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.401+0000 7f18db510640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d40b40c0 con 0x7f18d4143d10 2024-03-29T10:50:14.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.401+0000 7f18db510640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:14.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.401+0000 7f18db510640 10 monclient(hunting): authenticate will time out at 1027.642742s 2024-03-29T10:50:14.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.402+0000 7f18d92ac640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f18d40b1f60 0x7f18d4143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:42280/0 (socket says 172.21.15.5:42280) 2024-03-29T10:50:14.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.402+0000 7f18d92ac640 1 -- 172.21.15.5:0/2726016477 learned_addr learned my addr 172.21.15.5:0/2726016477 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.403+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725146335 0 0) 0x7f18d406ef80 con 0x7f18d40b1f60 2024-03-29T10:50:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.403+0000 7f18c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.403+0000 7f18c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.403+0000 7f18c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.403+0000 7f18c7fff640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:50:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.403+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c0003520 con 0x7f18d40b1f60 2024-03-29T10:50:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.403+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187741808 0 0) 0x7f18d406f260 con 0x7f18d4071eb0 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.403+0000 7f18c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d406ef80 con 0x7f18d4071eb0 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563145104 0 0) 0x7f18d40b40c0 con 0x7f18d4143d10 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d406f260 con 0x7f18d4143d10 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 989490802 0 0) 0x7f18c0003520 con 0x7f18d40b1f60 2024-03-29T10:50:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 >> v1:172.21.15.5:6790/0 conn(0x7f18d4143d10 legacy=0x7f18d4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.404+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 >> v1:172.21.15.177:6789/0 conn(0x7f18d4071eb0 legacy=0x7f18d4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 1 monclient: found mon.noname-a 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4155d70 con 0x7f18d40b1f60 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 10 monclient: _finish_auth 0 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f18cc001940 con 0x7f18d40b1f60 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 10 monclient: dump: 2024-03-29T10:50:14.391 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18db510640 5 monclient: authenticate success, global_id 4305 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18db510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18cc001b60 con 0x7f18d40b1f60 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18db510640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.405+0000 7f18db510640 4 set_mon_vals no callback set 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.406+0000 7f18c7fff640 1 -- 172.21.15.5:0/2726016477 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f18cc001ef0 con 0x7f18d40b1f60 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.406+0000 7f18c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.406+0000 7f18c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.406+0000 7f18c7fff640 10 monclient: dump: 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:14.393 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.406+0000 7f18db510640 10 monclient: shutdown 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.406+0000 7f18db510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.406+0000 7f18db510640 1 -- 172.21.15.5:0/2726016477 >> v1:172.21.15.5:6789/0 conn(0x7f18d40b1f60 legacy=0x7f18d4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.406+0000 7f18db510640 1 -- 172.21.15.5:0/2726016477 shutdown_connections 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.406+0000 7f18db510640 1 -- 172.21.15.5:0/2726016477 >> 172.21.15.5:0/2726016477 conn(0x7f18d4066a00 msgr2=0x7f18d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.407+0000 7f18db510640 1 -- 172.21.15.5:0/2726016477 shutdown_connections 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.407+0000 7f18db510640 1 -- 172.21.15.5:0/2726016477 wait complete. 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.408+0000 7f18db510640 10 monclient: build_initial_monmap 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.408+0000 7f18db510640 10 monclient: monmap: 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:50:14.394 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:50:14.395 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:14.395 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:14.395 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:14.395 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:50:14.395 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:50:14.395 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:14.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.409+0000 7f18db510640 1 librados: starting msgr at 2024-03-29T10:50:14.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.409+0000 7f18db510640 1 librados: starting objecter 2024-03-29T10:50:14.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.409+0000 7f18db510640 1 Processor -- start 2024-03-29T10:50:14.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.409+0000 7f18db510640 1 -- start start 2024-03-29T10:50:14.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.409+0000 7f18db510640 1 librados: setting wanted keys 2024-03-29T10:50:14.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.409+0000 7f18db510640 1 librados: calling monclient init 2024-03-29T10:50:14.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.409+0000 7f18db510640 10 monclient: init 2024-03-29T10:50:14.399 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.410+0000 7f18db510640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.410+0000 7f18db510640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.410+0000 7f18db510640 10 monclient(hunting): picked mon.noname-a con 0x7f18d40b1f60 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.410+0000 7f18db510640 10 monclient(hunting): picked mon.noname-b con 0x7f18d4143d10 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.410+0000 7f18db510640 10 monclient(hunting): picked mon.noname-c con 0x7f18d4132a30 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.410+0000 7f18db510640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d41550c0 con 0x7f18d40b1f60 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.410+0000 7f18db510640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d4133790 con 0x7f18d4143d10 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.410+0000 7f18db510640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d4070730 con 0x7f18d4132a30 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.410+0000 7f18db510640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.410+0000 7f18db510640 10 monclient(hunting): authenticate will time out at 1027.652070s 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18d8aab640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f18d4143d10 0x7f18d4131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:52494/0 (socket says 172.21.15.5:52494) 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18d8aab640 1 -- 172.21.15.5:0/2364155830 learned_addr learned my addr 172.21.15.5:0/2364155830 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18c5ffb640 10 client.?.objecter ms_handle_connect 0x7f18d4143d10 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18c5ffb640 10 client.?.objecter ms_handle_connect 0x7f18d4132a30 2024-03-29T10:50:14.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002197639 0 0) 0x7f18d4133790 con 0x7f18d4143d10 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18c5ffb640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.411+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18b8004eb0 con 0x7f18d4143d10 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 client.?.objecter ms_handle_connect 0x7f18d40b1f60 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926363338 0 0) 0x7f18d4070730 con 0x7f18d4132a30 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:14.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d4133790 con 0x7f18d4132a30 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221820803 0 0) 0x7f18b8004eb0 con 0x7f18d4143d10 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18d4070730 con 0x7f18d4143d10 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f18c80019d0 con 0x7f18d4143d10 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): dump: 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:14.402 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280196008 0 0) 0x7f18d41550c0 con 0x7f18d40b1f60 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.412+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18b8004eb0 con 0x7f18d40b1f60 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4253895307 0 0) 0x7f18d4133790 con 0x7f18d4132a30 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18d41550c0 con 0x7f18d4132a30 2024-03-29T10:50:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f18d0001810 con 0x7f18d4132a30 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 10 monclient(hunting): dump: 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:14.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2578440984 0 0) 0x7f18b8004eb0 con 0x7f18d40b1f60 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18d4133790 con 0x7f18d40b1f60 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f18cc001880 con 0x7f18d40b1f60 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 10 monclient(hunting): dump: 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:14.405 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2796689159 0 0) 0x7f18d4070730 con 0x7f18d4143d10 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 >> v1:172.21.15.5:6790/0 conn(0x7f18d4132a30 legacy=0x7f18d4132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 >> v1:172.21.15.5:6789/0 conn(0x7f18d40b1f60 legacy=0x7f18d4137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 monclient: found mon.b 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4132740 con 0x7f18d4143d10 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:14.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18db510640 5 monclient: authenticate success, global_id 4303 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18db510640 10 monclient: _renew_subs 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18db510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.413+0000 7f18db510640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d4070be0 con 0x7f18d4143d10 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18db510640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18db510640 10 monclient: _renew_subs 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18db510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18db510640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d4136c40 con 0x7f18d4143d10 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18db510640 1 librados: init done 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18c80035f0 con 0x7f18d4143d10 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f18c8003900 con 0x7f18d4143d10 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18c5ffb640 10 monclient: dump: 2024-03-29T10:50:14.407 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.414+0000 7f18c77fe640 4 set_mon_vals no callback set 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.415+0000 7f18db510640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:14.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.415+0000 7f18db510640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:14.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.415+0000 7f18db510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:50:14.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.415+0000 7f18db510640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b4005080 con 0x7f18d4143d10 2024-03-29T10:50:14.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.416+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f18c8003a60 con 0x7f18d4143d10 2024-03-29T10:50:14.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.417+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f18c8084a40 con 0x7f18d4143d10 2024-03-29T10:50:14.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.417+0000 7f18c5ffb640 10 client.4303.objecter ms_dispatch 0x7f18d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:14.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.417+0000 7f18c5ffb640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:14.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.417+0000 7f18c5ffb640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:14.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.417+0000 7f18c5ffb640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-29T10:50:14.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.417+0000 7f18d92ac640 10 monclient: get_auth_request con 0x7f18b806e1b0 auth_method 0 2024-03-29T10:50:14.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.417+0000 7f18c5ffb640 10 client.4303.objecter ms_handle_connect 0x7f18b806e1b0 2024-03-29T10:50:14.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.423+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18c8050150 con 0x7f18d4143d10 2024-03-29T10:50:14.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.423+0000 7f18c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:14.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.423+0000 7f18c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:14.775 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.793+0000 7f18db510640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:14.775 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.793+0000 7f18db510640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:14.775 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.793+0000 7f18db510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:50:14.776 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.793+0000 7f18db510640 1 -- 172.21.15.5:0/2364155830 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18b4005330 con 0x7f18d4143d10 2024-03-29T10:50:14.778 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.796+0000 7f18c5ffb640 1 -- 172.21.15.5:0/2364155830 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f18c8052250 con 0x7f18d4143d10 2024-03-29T10:50:14.778 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.797+0000 7f18c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:14.778 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.797+0000 7f18c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:14.779 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:50:14.779 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:14.782 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.800+0000 7f18db510640 10 librados: watch_flush enter 2024-03-29T10:50:14.782 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.801+0000 7f18db510640 10 librados: watch_flush exit 2024-03-29T10:50:14.782 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.801+0000 7f18db510640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-29T10:50:14.783 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.801+0000 7f18db510640 10 client.4303.objecter successfully canceled tick 2024-03-29T10:50:14.783 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.801+0000 7f18db510640 1 -- 172.21.15.5:0/2364155830 >> v1:172.21.15.5:6816/2847500151 conn(0x7f18b806e1b0 legacy=0x7f18b8070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:14.783 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.801+0000 7f18db510640 10 monclient: shutdown 2024-03-29T10:50:14.783 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.801+0000 7f18db510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:14.784 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.801+0000 7f18db510640 1 -- 172.21.15.5:0/2364155830 >> v1:172.21.15.177:6789/0 conn(0x7f18d4143d10 legacy=0x7f18d4131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:14.784 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.801+0000 7f18db510640 1 -- 172.21.15.5:0/2364155830 shutdown_connections 2024-03-29T10:50:14.784 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.801+0000 7f18db510640 1 -- 172.21.15.5:0/2364155830 >> 172.21.15.5:0/2364155830 conn(0x7f18d4066a00 msgr2=0x7f18d4146ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:14.784 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.802+0000 7f18db510640 1 -- 172.21.15.5:0/2364155830 shutdown_connections 2024-03-29T10:50:14.784 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.802+0000 7f18db510640 1 -- 172.21.15.5:0/2364155830 wait complete. 2024-03-29T10:50:14.784 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:14.802+0000 7f18db510640 1 librados: shutdown 2024-03-29T10:50:14.798 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:21.605 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.749+0000 7f06faaae640 10 monclient: get_monmap_and_config 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.749+0000 7f06faaae640 10 monclient: build_initial_monmap 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.749+0000 7f06faaae640 10 monclient: monmap: 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:50:21.750139+0000 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:50:21.750139+0000 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:50:21.731 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:21.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.749+0000 7f06faaae640 1 Processor -- start 2024-03-29T10:50:21.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.749+0000 7f06faaae640 1 -- start start 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.750+0000 7f06faaae640 10 monclient: init 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.750+0000 7f06faaae640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.750+0000 7f06faaae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.750+0000 7f06faaae640 10 monclient(hunting): picked mon.noname-c con 0x7f06f41457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06faaae640 10 monclient(hunting): picked mon.noname-b con 0x7f06f413f650 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06faaae640 10 monclient(hunting): picked mon.noname-a con 0x7f06f40a9d70 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06faaae640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f40aaef0 con 0x7f06f40a9d70 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06faaae640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f4145bd0 con 0x7f06f41457f0 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06faaae640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f4145eb0 con 0x7f06f413f650 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06faaae640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06faaae640 10 monclient(hunting): authenticate will time out at 1034.992324s 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f3fff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f06f413f650 0x7f06f40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:38896/0 (socket says 172.21.15.5:38896) 2024-03-29T10:50:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f3fff640 1 -- 172.21.15.5:0/1805056325 learned_addr learned my addr 172.21.15.5:0/1805056325 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:21.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623539869 0 0) 0x7f06f4145eb0 con 0x7f06f413f650 2024-03-29T10:50:21.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:21.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:21.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:21.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:50:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e4003520 con 0x7f06f413f650 2024-03-29T10:50:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720907531 0 0) 0x7f06f4145bd0 con 0x7f06f41457f0 2024-03-29T10:50:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:50:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.751+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f4145eb0 con 0x7f06f41457f0 2024-03-29T10:50:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.752+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4280987446 0 0) 0x7f06e4003520 con 0x7f06f413f650 2024-03-29T10:50:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.752+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 >> v1:172.21.15.177:6789/0 conn(0x7f06f41457f0 legacy=0x7f06f413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.752+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 >> v1:172.21.15.5:6789/0 conn(0x7f06f40a9d70 legacy=0x7f06f414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 1 monclient: found mon.noname-b 2024-03-29T10:50:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:50:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f4146340 con 0x7f06f413f650 2024-03-29T10:50:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f06e0001690 con 0x7f06f413f650 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: dump: 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06faaae640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:50:21.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06faaae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06e00017f0 con 0x7f06f413f650 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 1 -- 172.21.15.5:0/1805056325 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f06e0001c00 con 0x7f06f413f650 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06faaae640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06faaae640 4 set_mon_vals no callback set 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.754+0000 7f06f37fe640 10 monclient: dump: 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:21.738 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:21.739 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:21.739 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:21.739 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:21.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.755+0000 7f06faaae640 10 monclient: shutdown 2024-03-29T10:50:21.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.755+0000 7f06faaae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:21.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.755+0000 7f06faaae640 1 -- 172.21.15.5:0/1805056325 >> v1:172.21.15.5:6790/0 conn(0x7f06f413f650 legacy=0x7f06f40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:21.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.755+0000 7f06faaae640 1 -- 172.21.15.5:0/1805056325 shutdown_connections 2024-03-29T10:50:21.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.755+0000 7f06faaae640 1 -- 172.21.15.5:0/1805056325 >> 172.21.15.5:0/1805056325 conn(0x7f06f4066a00 msgr2=0x7f06f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.755+0000 7f06faaae640 1 -- 172.21.15.5:0/1805056325 shutdown_connections 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.755+0000 7f06faaae640 1 -- 172.21.15.5:0/1805056325 wait complete. 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.756+0000 7f06faaae640 10 monclient: build_initial_monmap 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.756+0000 7f06faaae640 10 monclient: monmap: 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:50:21.747 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.756+0000 7f06faaae640 1 librados: starting msgr at 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.756+0000 7f06faaae640 1 librados: starting objecter 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.756+0000 7f06faaae640 1 Processor -- start 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.756+0000 7f06faaae640 1 -- start start 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.756+0000 7f06faaae640 1 librados: setting wanted keys 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.756+0000 7f06faaae640 1 librados: calling monclient init 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.756+0000 7f06faaae640 10 monclient: init 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06faaae640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06faaae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06faaae640 10 monclient(hunting): picked mon.noname-b con 0x7f06f40a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:21.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06faaae640 10 monclient(hunting): picked mon.noname-a con 0x7f06f413f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06faaae640 10 monclient(hunting): picked mon.noname-c con 0x7f06f41457f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06faaae640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f41da680 con 0x7f06f413f650 2024-03-29T10:50:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06faaae640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f413b5d0 con 0x7f06f40a9d70 2024-03-29T10:50:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06faaae640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f41db500 con 0x7f06f41457f0 2024-03-29T10:50:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06faaae640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06faaae640 10 monclient(hunting): authenticate will time out at 1034.998853s 2024-03-29T10:50:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06f904b640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f06f41457f0 0x7f06f41e1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:39132/0 (socket says 172.21.15.5:39132) 2024-03-29T10:50:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.757+0000 7f06f904b640 1 -- 172.21.15.5:0/2825960302 learned_addr learned my addr 172.21.15.5:0/2825960302 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 client.?.objecter ms_handle_connect 0x7f06f41457f0 2024-03-29T10:50:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 client.?.objecter ms_handle_connect 0x7f06f40a9d70 2024-03-29T10:50:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743820403 0 0) 0x7f06f41db500 con 0x7f06f41457f0 2024-03-29T10:50:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:50:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d8004eb0 con 0x7f06f41457f0 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462254278 0 0) 0x7f06f413b5d0 con 0x7f06f40a9d70 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.758+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f41db500 con 0x7f06f40a9d70 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2018723832 0 0) 0x7f06d8004eb0 con 0x7f06f41457f0 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06d8005490 con 0x7f06f41457f0 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f06ec001c60 con 0x7f06f41457f0 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 10 monclient(hunting): dump: 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4132929019 0 0) 0x7f06f41db500 con 0x7f06f40a9d70 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06d8004eb0 con 0x7f06f40a9d70 2024-03-29T10:50:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f06e8001840 con 0x7f06f40a9d70 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 10 monclient(hunting): dump: 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4082489153 0 0) 0x7f06d8005490 con 0x7f06f41457f0 2024-03-29T10:50:21.753 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 >> v1:172.21.15.177:6789/0 conn(0x7f06f40a9d70 legacy=0x7f06f413aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.759+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 >> v1:172.21.15.5:6789/0 conn(0x7f06f413f650 legacy=0x7f06f41dadf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 1 monclient: found mon.c 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f413aaa0 con 0x7f06f41457f0 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 10 monclient: _finish_auth 0 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06faaae640 5 monclient: authenticate success, global_id 4343 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06faaae640 10 monclient: _renew_subs 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06faaae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06faaae640 1 -- 172.21.15.5:0/2825960302 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f41da8d0 con 0x7f06f41457f0 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06ec003970 con 0x7f06f41457f0 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f06ec003d40 con 0x7f06f41457f0 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f17fa640 10 monclient: dump: 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:21.755 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06faaae640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06faaae640 10 monclient: _renew_subs 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06faaae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06faaae640 1 -- 172.21.15.5:0/2825960302 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f4142c50 con 0x7f06f41457f0 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06f2ffd640 4 set_mon_vals no callback set 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.761+0000 7f06faaae640 1 librados: init done 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.762+0000 7f06faaae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.762+0000 7f06faaae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.762+0000 7f06faaae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.763+0000 7f06faaae640 1 -- 172.21.15.5:0/2825960302 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f41dab10 con 0x7f06f41457f0 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.763+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f06ec003ea0 con 0x7f06f41457f0 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.764+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f06ec084cc0 con 0x7f06f41457f0 2024-03-29T10:50:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.764+0000 7f06f17fa640 10 client.4343.objecter ms_dispatch 0x7f06f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:21.757 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.764+0000 7f06f17fa640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:21.757 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.764+0000 7f06f17fa640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:21.757 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.764+0000 7f06f17fa640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T10:50:21.757 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.764+0000 7f06f884a640 10 monclient: get_auth_request con 0x7f06d806e1b0 auth_method 0 2024-03-29T10:50:21.757 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.765+0000 7f06f17fa640 10 client.4343.objecter ms_handle_connect 0x7f06d806e1b0 2024-03-29T10:50:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.772+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06f41dab10 con 0x7f06f41457f0 2024-03-29T10:50:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.772+0000 7f06f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:21.772+0000 7f06f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:22.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.054+0000 7f06faaae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:22.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.054+0000 7f06faaae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:22.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.054+0000 7f06faaae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:22.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.054+0000 7f06faaae640 1 -- 172.21.15.5:0/2825960302 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06f41dab10 con 0x7f06f41457f0 2024-03-29T10:50:22.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.055+0000 7f06f17fa640 1 -- 172.21.15.5:0/2825960302 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f06f41dab10 con 0x7f06f41457f0 2024-03-29T10:50:22.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.055+0000 7f06f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:22.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.055+0000 7f06f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:22.038 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:50:22.039 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:22.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 10 librados: watch_flush enter 2024-03-29T10:50:22.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 10 librados: watch_flush exit 2024-03-29T10:50:22.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T10:50:22.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 10 client.4343.objecter successfully canceled tick 2024-03-29T10:50:22.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 1 -- 172.21.15.5:0/2825960302 >> v1:172.21.15.5:6816/2847500151 conn(0x7f06d806e1b0 legacy=0x7f06d8070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:22.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 10 monclient: shutdown 2024-03-29T10:50:22.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:22.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 1 -- 172.21.15.5:0/2825960302 >> v1:172.21.15.5:6790/0 conn(0x7f06f41457f0 legacy=0x7f06f41e1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:22.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 1 -- 172.21.15.5:0/2825960302 shutdown_connections 2024-03-29T10:50:22.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 1 -- 172.21.15.5:0/2825960302 >> 172.21.15.5:0/2825960302 conn(0x7f06f4066a00 msgr2=0x7f06f4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:22.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 1 -- 172.21.15.5:0/2825960302 shutdown_connections 2024-03-29T10:50:22.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 1 -- 172.21.15.5:0/2825960302 wait complete. 2024-03-29T10:50:22.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:22.058+0000 7f06faaae640 1 librados: shutdown 2024-03-29T10:50:22.053 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:28.858 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:28.984 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.002+0000 7f2c2f77a640 10 monclient: get_monmap_and_config 2024-03-29T10:50:28.984 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.002+0000 7f2c2f77a640 10 monclient: build_initial_monmap 2024-03-29T10:50:28.984 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.002+0000 7f2c2f77a640 10 monclient: monmap: 2024-03-29T10:50:28.984 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:28.984 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:28.984 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:50:29.003603+0000 2024-03-29T10:50:28.984 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:50:29.003603+0000 2024-03-29T10:50:28.985 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:28.985 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:28.985 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:28.985 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:50:28.985 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:50:28.985 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:28.986 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.003+0000 7f2c2f77a640 1 Processor -- start 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.003+0000 7f2c2f77a640 1 -- start start 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.003+0000 7f2c2f77a640 10 monclient: init 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2f77a640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2f77a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2f77a640 10 monclient(hunting): picked mon.noname-c con 0x7f2c281457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2f77a640 10 monclient(hunting): picked mon.noname-a con 0x7f2c2813f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2f77a640 10 monclient(hunting): picked mon.noname-b con 0x7f2c280a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2f77a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c280aaef0 con 0x7f2c2813f650 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2f77a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c28145bd0 con 0x7f2c281457f0 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2f77a640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c28145eb0 con 0x7f2c280a9d70 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2f77a640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2f77a640 10 monclient(hunting): authenticate will time out at 1042.245515s 2024-03-29T10:50:28.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2dd17640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f2c280a9d70 0x7f2c2814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:59378/0 (socket says 172.21.15.5:59378) 2024-03-29T10:50:28.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.004+0000 7f2c2dd17640 1 -- 172.21.15.5:0/1269214776 learned_addr learned my addr 172.21.15.5:0/1269214776 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:28.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.005+0000 7f2c1ffff640 1 -- 172.21.15.5:0/1269214776 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9568002 0 0) 0x7f2c28145eb0 con 0x7f2c280a9d70 2024-03-29T10:50:28.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.005+0000 7f2c1ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:28.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.005+0000 7f2c1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:28.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.005+0000 7f2c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:28.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.005+0000 7f2c1ffff640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:50:28.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.005+0000 7f2c1ffff640 1 -- 172.21.15.5:0/1269214776 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c14003520 con 0x7f2c280a9d70 2024-03-29T10:50:28.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.005+0000 7f2c1ffff640 1 -- 172.21.15.5:0/1269214776 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 293615264 0 0) 0x7f2c14003520 con 0x7f2c280a9d70 2024-03-29T10:50:28.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.005+0000 7f2c1ffff640 1 -- 172.21.15.5:0/1269214776 >> v1:172.21.15.177:6789/0 conn(0x7f2c281457f0 legacy=0x7f2c2813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:50:28.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.005+0000 7f2c1ffff640 1 -- 172.21.15.5:0/1269214776 >> v1:172.21.15.5:6789/0 conn(0x7f2c2813f650 legacy=0x7f2c280b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:28.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:28.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 1 monclient: found mon.noname-b 2024-03-29T10:50:28.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:28.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:50:28.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 1 -- 172.21.15.5:0/1269214776 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c28146340 con 0x7f2c280a9d70 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 10 monclient: _finish_auth 0 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 1 -- 172.21.15.5:0/1269214776 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2c200019f0 con 0x7f2c280a9d70 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 10 monclient: dump: 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:28.992 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c2f77a640 5 monclient: authenticate success, global_id 4346 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c2f77a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 1 -- 172.21.15.5:0/1269214776 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c20001c50 con 0x7f2c280a9d70 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c2f77a640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.008+0000 7f2c2f77a640 4 set_mon_vals no callback set 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.009+0000 7f2c1ffff640 1 -- 172.21.15.5:0/1269214776 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2c200020a0 con 0x7f2c280a9d70 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.009+0000 7f2c1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.009+0000 7f2c1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.009+0000 7f2c1ffff640 10 monclient: dump: 2024-03-29T10:50:28.993 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.009+0000 7f2c2f77a640 10 monclient: shutdown 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.009+0000 7f2c2f77a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.009+0000 7f2c2f77a640 1 -- 172.21.15.5:0/1269214776 >> v1:172.21.15.5:6790/0 conn(0x7f2c280a9d70 legacy=0x7f2c2814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.009+0000 7f2c2f77a640 1 -- 172.21.15.5:0/1269214776 shutdown_connections 2024-03-29T10:50:28.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.009+0000 7f2c2f77a640 1 -- 172.21.15.5:0/1269214776 >> 172.21.15.5:0/1269214776 conn(0x7f2c28066a00 msgr2=0x7f2c28066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:28.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.009+0000 7f2c2f77a640 1 -- 172.21.15.5:0/1269214776 shutdown_connections 2024-03-29T10:50:28.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.010+0000 7f2c2f77a640 1 -- 172.21.15.5:0/1269214776 wait complete. 2024-03-29T10:50:29.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.010+0000 7f2c2f77a640 10 monclient: build_initial_monmap 2024-03-29T10:50:29.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.010+0000 7f2c2f77a640 10 monclient: monmap: 2024-03-29T10:50:29.000 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:29.000 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:29.000 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:50:29.000 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.011+0000 7f2c2f77a640 1 librados: starting msgr at 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.011+0000 7f2c2f77a640 1 librados: starting objecter 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.011+0000 7f2c2f77a640 1 Processor -- start 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.011+0000 7f2c2f77a640 1 -- start start 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.011+0000 7f2c2f77a640 1 librados: setting wanted keys 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.011+0000 7f2c2f77a640 1 librados: calling monclient init 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.011+0000 7f2c2f77a640 10 monclient: init 2024-03-29T10:50:29.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2f77a640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2f77a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2f77a640 10 monclient(hunting): picked mon.noname-b con 0x7f2c280a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2f77a640 10 monclient(hunting): picked mon.noname-c con 0x7f2c2813f650 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2f77a640 10 monclient(hunting): picked mon.noname-a con 0x7f2c281457f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2cd15640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f2c2813f650 0x7f2c281df180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:59650/0 (socket says 172.21.15.5:59650) 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2cd15640 1 -- 172.21.15.5:0/2048117502 learned_addr learned my addr 172.21.15.5:0/2048117502 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c1dffb640 10 client.?.objecter ms_handle_connect 0x7f2c2813f650 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c1dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2813bc00 con 0x7f2c281457f0 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2813ac10 con 0x7f2c280a9d70 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2813ada0 con 0x7f2c2813f650 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.013+0000 7f2c2f77a640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c2f77a640 10 monclient(hunting): authenticate will time out at 1042.255302s 2024-03-29T10:50:29.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979356879 0 0) 0x7f2c2813ada0 con 0x7f2c2813f650 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c0c004eb0 con 0x7f2c2813f650 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 10 client.?.objecter ms_handle_connect 0x7f2c280a9d70 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1241761809 0 0) 0x7f2c0c004eb0 con 0x7f2c2813f650 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c2813ada0 con 0x7f2c2813f650 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2c18004080 con 0x7f2c2813f650 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 10 monclient(hunting): dump: 2024-03-29T10:50:29.003 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.014+0000 7f2c1dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228485925 0 0) 0x7f2c2813ac10 con 0x7f2c280a9d70 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:29.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c0c004eb0 con 0x7f2c280a9d70 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1431606333 0 0) 0x7f2c2813ada0 con 0x7f2c2813f650 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 >> v1:172.21.15.177:6789/0 conn(0x7f2c280a9d70 legacy=0x7f2c281dea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 >> v1:172.21.15.5:6789/0 conn(0x7f2c281457f0 legacy=0x7f2c281e5f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 1 monclient: found mon.c 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c28142940 con 0x7f2c2813f650 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:29.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c2f77a640 5 monclient: authenticate success, global_id 4349 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c2f77a640 10 monclient: _renew_subs 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c2f77a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c28142b30 con 0x7f2c2813f650 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c180042a0 con 0x7f2c2813f650 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.015+0000 7f2c1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.016+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2c18001320 con 0x7f2c2813f650 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.016+0000 7f2c1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.016+0000 7f2c1dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.016+0000 7f2c1dffb640 10 monclient: dump: 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:29.006 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.016+0000 7f2c2f77a640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.016+0000 7f2c2f77a640 10 monclient: _renew_subs 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.016+0000 7f2c2f77a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.016+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2806e070 con 0x7f2c2813f650 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.016+0000 7f2c2f77a640 1 librados: init done 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.016+0000 7f2c1f7fe640 4 set_mon_vals no callback set 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.017+0000 7f2c2f77a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.017+0000 7f2c2f77a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.017+0000 7f2c2f77a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.017+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c08005080 con 0x7f2c2813f650 2024-03-29T10:50:29.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.018+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f2c18001500 con 0x7f2c2813f650 2024-03-29T10:50:29.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.019+0000 7f2c2d516640 10 monclient: get_auth_request con 0x7f2c0c06dcb0 auth_method 0 2024-03-29T10:50:29.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.019+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f2c18084bb0 con 0x7f2c2813f650 2024-03-29T10:50:29.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.019+0000 7f2c1dffb640 10 client.4349.objecter ms_dispatch 0x7f2c28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:29.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.020+0000 7f2c1dffb640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:29.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.020+0000 7f2c1dffb640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:29.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.020+0000 7f2c1dffb640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T10:50:29.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.020+0000 7f2c1dffb640 10 client.4349.objecter ms_handle_connect 0x7f2c0c06dcb0 2024-03-29T10:50:29.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.024+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c180504f0 con 0x7f2c2813f650 2024-03-29T10:50:29.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.024+0000 7f2c1dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:29.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.024+0000 7f2c1dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:29.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.297+0000 7f2c2f77a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:29.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.297+0000 7f2c2f77a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:29.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.297+0000 7f2c2f77a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:29.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.297+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c08005330 con 0x7f2c2813f650 2024-03-29T10:50:29.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.298+0000 7f2c1dffb640 1 -- 172.21.15.5:0/2048117502 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f2c18084260 con 0x7f2c2813f650 2024-03-29T10:50:29.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.298+0000 7f2c1dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:29.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.298+0000 7f2c1dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:29.280 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:50:29.280 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:29.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 10 librados: watch_flush enter 2024-03-29T10:50:29.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 10 librados: watch_flush exit 2024-03-29T10:50:29.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T10:50:29.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 10 client.4349.objecter successfully canceled tick 2024-03-29T10:50:29.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 >> v1:172.21.15.5:6816/2847500151 conn(0x7f2c0c06dcb0 legacy=0x7f2c0c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:29.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 10 monclient: shutdown 2024-03-29T10:50:29.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:29.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 >> v1:172.21.15.5:6790/0 conn(0x7f2c2813f650 legacy=0x7f2c281df180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:29.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 shutdown_connections 2024-03-29T10:50:29.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 >> 172.21.15.5:0/2048117502 conn(0x7f2c28066a00 msgr2=0x7f2c280b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:29.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 shutdown_connections 2024-03-29T10:50:29.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.301+0000 7f2c2f77a640 1 -- 172.21.15.5:0/2048117502 wait complete. 2024-03-29T10:50:29.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:29.302+0000 7f2c2f77a640 1 librados: shutdown 2024-03-29T10:50:29.295 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:36.101 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:36.243 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.261+0000 7f2eeadd0640 10 monclient: get_monmap_and_config 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.261+0000 7f2eeadd0640 10 monclient: build_initial_monmap 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.261+0000 7f2eeadd0640 10 monclient: monmap: 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:50:36.262157+0000 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:50:36.262157+0000 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:50:36.244 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:36.246 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.262+0000 7f2eeadd0640 1 Processor -- start 2024-03-29T10:50:36.246 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 1 -- start start 2024-03-29T10:50:36.246 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 10 monclient: init 2024-03-29T10:50:36.246 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee40acf60 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee413ff40 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee41403f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40b4310 con 0x7f2ee41403f0 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee4140800 con 0x7f2ee413ff40 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40aec00 con 0x7f2ee40acf60 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2eeadd0640 10 monclient(hunting): authenticate will time out at 1049.504861s 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.263+0000 7f2ee8b6c640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f2ee40acf60 0x7f2ee4139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:57924/0 (socket says 172.21.15.5:57924) 2024-03-29T10:50:36.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.264+0000 7f2ee8b6c640 1 -- 172.21.15.5:0/3205007080 learned_addr learned my addr 172.21.15.5:0/3205007080 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:36.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.266+0000 7f2ee37fe640 1 -- 172.21.15.5:0/3205007080 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066539051 0 0) 0x7f2ee40aec00 con 0x7f2ee40acf60 2024-03-29T10:50:36.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.266+0000 7f2ee37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:36.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.266+0000 7f2ee37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:36.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.266+0000 7f2ee37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:36.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.266+0000 7f2ee37fe640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:50:36.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.266+0000 7f2ee37fe640 1 -- 172.21.15.5:0/3205007080 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed0003520 con 0x7f2ee40acf60 2024-03-29T10:50:36.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.266+0000 7f2ee37fe640 1 -- 172.21.15.5:0/3205007080 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4232151611 0 0) 0x7f2ed0003520 con 0x7f2ee40acf60 2024-03-29T10:50:36.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.266+0000 7f2ee37fe640 1 -- 172.21.15.5:0/3205007080 >> v1:172.21.15.177:6789/0 conn(0x7f2ee413ff40 legacy=0x7f2ee4148840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:36.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.266+0000 7f2ee37fe640 1 -- 172.21.15.5:0/3205007080 >> v1:172.21.15.5:6789/0 conn(0x7f2ee41403f0 legacy=0x7f2ee414bf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:36.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:36.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 1 monclient: found mon.noname-b 2024-03-29T10:50:36.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:36.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:50:36.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 1 -- 172.21.15.5:0/3205007080 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee40aeee0 con 0x7f2ee40acf60 2024-03-29T10:50:36.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:36.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:36.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 1 -- 172.21.15.5:0/3205007080 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2edc001690 con 0x7f2ee40acf60 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2ee37fe640 10 monclient: dump: 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:36.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2eeadd0640 5 monclient: authenticate success, global_id 4352 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.267+0000 7f2eeadd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.268+0000 7f2ee37fe640 1 -- 172.21.15.5:0/3205007080 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2edc001920 con 0x7f2ee40acf60 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.268+0000 7f2ee37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.268+0000 7f2ee37fe640 1 -- 172.21.15.5:0/3205007080 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2edc001cb0 con 0x7f2ee40acf60 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.268+0000 7f2ee37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.268+0000 7f2ee37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.268+0000 7f2ee37fe640 10 monclient: dump: 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:36.253 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:36.254 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:36.254 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:36.254 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:36.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.268+0000 7f2eeadd0640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:36.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.268+0000 7f2eeadd0640 4 set_mon_vals no callback set 2024-03-29T10:50:36.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.269+0000 7f2eeadd0640 10 monclient: shutdown 2024-03-29T10:50:36.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.269+0000 7f2eeadd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:36.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.269+0000 7f2eeadd0640 1 -- 172.21.15.5:0/3205007080 >> v1:172.21.15.5:6790/0 conn(0x7f2ee40acf60 legacy=0x7f2ee4139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:36.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.269+0000 7f2eeadd0640 1 -- 172.21.15.5:0/3205007080 shutdown_connections 2024-03-29T10:50:36.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.269+0000 7f2eeadd0640 1 -- 172.21.15.5:0/3205007080 >> 172.21.15.5:0/3205007080 conn(0x7f2ee40b35e0 msgr2=0x7f2ee40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:36.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.270+0000 7f2eeadd0640 1 -- 172.21.15.5:0/3205007080 shutdown_connections 2024-03-29T10:50:36.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.271+0000 7f2eeadd0640 1 -- 172.21.15.5:0/3205007080 wait complete. 2024-03-29T10:50:36.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.271+0000 7f2eeadd0640 10 monclient: build_initial_monmap 2024-03-29T10:50:36.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.271+0000 7f2eeadd0640 10 monclient: monmap: 2024-03-29T10:50:36.256 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.272+0000 7f2eeadd0640 1 librados: starting msgr at 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.272+0000 7f2eeadd0640 1 librados: starting objecter 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.272+0000 7f2eeadd0640 1 Processor -- start 2024-03-29T10:50:36.257 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.272+0000 7f2eeadd0640 1 -- start start 2024-03-29T10:50:36.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.272+0000 7f2eeadd0640 1 librados: setting wanted keys 2024-03-29T10:50:36.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.272+0000 7f2eeadd0640 1 librados: calling monclient init 2024-03-29T10:50:36.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.272+0000 7f2eeadd0640 10 monclient: init 2024-03-29T10:50:36.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.273+0000 7f2eeadd0640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:36.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.273+0000 7f2eeadd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:36.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.273+0000 7f2eeadd0640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee413ff40 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:36.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.273+0000 7f2eeadd0640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee41403f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:36.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.273+0000 7f2eeadd0640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee4071a40 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:36.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.273+0000 7f2eeadd0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee412b260 con 0x7f2ee4071a40 2024-03-29T10:50:36.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.273+0000 7f2eeadd0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee412af30 con 0x7f2ee41403f0 2024-03-29T10:50:36.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.273+0000 7f2eeadd0640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee4127f90 con 0x7f2ee413ff40 2024-03-29T10:50:36.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.273+0000 7f2eeadd0640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:36.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.273+0000 7f2eeadd0640 10 monclient(hunting): authenticate will time out at 1049.514732s 2024-03-29T10:50:36.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.275+0000 7f2ee8b6c640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f2ee413ff40 0x7f2ee414ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:58134/0 (socket says 172.21.15.5:58134) 2024-03-29T10:50:36.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.275+0000 7f2ee8b6c640 1 -- 172.21.15.5:0/2167034533 learned_addr learned my addr 172.21.15.5:0/2167034533 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:36.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.275+0000 7f2ee17fa640 10 client.?.objecter ms_handle_connect 0x7f2ee413ff40 2024-03-29T10:50:36.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.276+0000 7f2ee17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:36.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.276+0000 7f2ee17fa640 10 client.?.objecter ms_handle_connect 0x7f2ee4071a40 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.276+0000 7f2ee17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.276+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266688704 0 0) 0x7f2ee412b260 con 0x7f2ee4071a40 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.276+0000 7f2ee17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.276+0000 7f2ee17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.276+0000 7f2ee17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.276+0000 7f2ee17fa640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.276+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec4004eb0 con 0x7f2ee4071a40 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1613981224 0 0) 0x7f2ec4004eb0 con 0x7f2ee4071a40 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ee412b260 con 0x7f2ee4071a40 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2ed4001ad0 con 0x7f2ee4071a40 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:36.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient(hunting): dump: 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 client.?.objecter ms_handle_connect 0x7f2ee41403f0 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:36.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 61230810 0 0) 0x7f2ee412b260 con 0x7f2ee4071a40 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 >> v1:172.21.15.5:6790/0 conn(0x7f2ee413ff40 legacy=0x7f2ee414ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 >> v1:172.21.15.177:6789/0 conn(0x7f2ee41403f0 legacy=0x7f2ee413ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 1 monclient: found mon.a 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee40724e0 con 0x7f2ee4071a40 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient: _finish_auth 0 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2eeadd0640 5 monclient: authenticate success, global_id 4311 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2eeadd0640 10 monclient: _renew_subs 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2eeadd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2eeadd0640 1 -- 172.21.15.5:0/2167034533 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee41e3850 con 0x7f2ee4071a40 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ed4002060 con 0x7f2ee4071a40 2024-03-29T10:50:36.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2ed4003b10 con 0x7f2ee4071a40 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee17fa640 10 monclient: dump: 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:36.269 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2eeadd0640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2ee2ffd640 4 set_mon_vals no callback set 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2eeadd0640 10 monclient: _renew_subs 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2eeadd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2eeadd0640 1 -- 172.21.15.5:0/2167034533 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee41e3bc0 con 0x7f2ee4071a40 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.277+0000 7f2eeadd0640 1 librados: init done 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.279+0000 7f2eeadd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:36.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.279+0000 7f2eeadd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:36.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.279+0000 7f2eeadd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:36.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.279+0000 7f2eeadd0640 1 -- 172.21.15.5:0/2167034533 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee413d630 con 0x7f2ee4071a40 2024-03-29T10:50:36.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.279+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f2ed4019ae0 con 0x7f2ee4071a40 2024-03-29T10:50:36.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.280+0000 7f2ee8b6c640 10 monclient: get_auth_request con 0x7f2ec406ddd0 auth_method 0 2024-03-29T10:50:36.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.281+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f2ed4083a80 con 0x7f2ee4071a40 2024-03-29T10:50:36.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.281+0000 7f2ee17fa640 10 client.4311.objecter ms_dispatch 0x7f2ee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:36.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.281+0000 7f2ee17fa640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:36.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.281+0000 7f2ee17fa640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:36.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.281+0000 7f2ee17fa640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T10:50:36.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.281+0000 7f2ee17fa640 10 client.4311.objecter ms_handle_connect 0x7f2ec406ddd0 2024-03-29T10:50:36.272 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.288+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ed4085e60 con 0x7f2ee4071a40 2024-03-29T10:50:36.272 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.288+0000 7f2ee17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:36.272 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.288+0000 7f2ee17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:36.584 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.602+0000 7f2eeadd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:36.584 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.602+0000 7f2eeadd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:36.584 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.602+0000 7f2eeadd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:36.584 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.602+0000 7f2eeadd0640 1 -- 172.21.15.5:0/2167034533 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ee412b0c0 con 0x7f2ee4071a40 2024-03-29T10:50:36.585 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.603+0000 7f2ee17fa640 1 -- 172.21.15.5:0/2167034533 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f2ee412b0c0 con 0x7f2ee4071a40 2024-03-29T10:50:36.585 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.603+0000 7f2ee17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:36.585 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.603+0000 7f2ee17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:36.585 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:50:36.585 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:36.588 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.607+0000 7f2eeadd0640 10 librados: watch_flush enter 2024-03-29T10:50:36.590 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 10 librados: watch_flush exit 2024-03-29T10:50:36.590 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T10:50:36.590 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 10 client.4311.objecter successfully canceled tick 2024-03-29T10:50:36.590 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 1 -- 172.21.15.5:0/2167034533 >> v1:172.21.15.5:6816/2847500151 conn(0x7f2ec406ddd0 legacy=0x7f2ec4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:36.590 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 10 monclient: shutdown 2024-03-29T10:50:36.590 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:36.590 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 1 -- 172.21.15.5:0/2167034533 >> v1:172.21.15.5:6789/0 conn(0x7f2ee4071a40 legacy=0x7f2ee4129fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:36.590 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 1 -- 172.21.15.5:0/2167034533 shutdown_connections 2024-03-29T10:50:36.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 1 -- 172.21.15.5:0/2167034533 >> 172.21.15.5:0/2167034533 conn(0x7f2ee40b35e0 msgr2=0x7f2ee40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:36.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 1 -- 172.21.15.5:0/2167034533 shutdown_connections 2024-03-29T10:50:36.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 1 -- 172.21.15.5:0/2167034533 wait complete. 2024-03-29T10:50:36.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:36.608+0000 7f2eeadd0640 1 librados: shutdown 2024-03-29T10:50:36.606 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:43.413 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.573+0000 7fae4b5e9640 10 monclient: get_monmap_and_config 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.573+0000 7fae4b5e9640 10 monclient: build_initial_monmap 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.573+0000 7fae4b5e9640 10 monclient: monmap: 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:50:43.574415+0000 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:50:43.574415+0000 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:50:43.555 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:50:43.556 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:43.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.575+0000 7fae4b5e9640 1 Processor -- start 2024-03-29T10:50:43.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.577+0000 7fae4b5e9640 1 -- start start 2024-03-29T10:50:43.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.577+0000 7fae4b5e9640 10 monclient: init 2024-03-29T10:50:43.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.578+0000 7fae4b5e9640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:43.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae4b5e9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:50:43.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae4b5e9640 10 monclient(hunting): picked mon.noname-a con 0x7fae441457f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:43.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae4b5e9640 10 monclient(hunting): picked mon.noname-b con 0x7fae4413f650 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:43.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae4b5e9640 10 monclient(hunting): picked mon.noname-c con 0x7fae440a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:43.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae4b5e9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae440aaef0 con 0x7fae441457f0 2024-03-29T10:50:43.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae4b5e9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae44145bd0 con 0x7fae440a9d70 2024-03-29T10:50:43.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae4b5e9640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae44145eb0 con 0x7fae4413f650 2024-03-29T10:50:43.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae4b5e9640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:43.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae4b5e9640 10 monclient(hunting): authenticate will time out at 1056.820203s 2024-03-29T10:50:43.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae48b84640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fae4413f650 0x7fae440b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:56282/0 (socket says 172.21.15.5:56282) 2024-03-29T10:50:43.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.579+0000 7fae48b84640 1 -- 172.21.15.5:0/517351204 learned_addr learned my addr 172.21.15.5:0/517351204 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:43.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772616736 0 0) 0x7fae440aaef0 con 0x7fae441457f0 2024-03-29T10:50:43.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:43.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:43.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:43.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:50:43.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae30003520 con 0x7fae441457f0 2024-03-29T10:50:43.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813469359 0 0) 0x7fae44145eb0 con 0x7fae4413f650 2024-03-29T10:50:43.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:43.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:43.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:43.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:50:43.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae300048a0 con 0x7fae4413f650 2024-03-29T10:50:43.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2596384203 0 0) 0x7fae30003520 con 0x7fae441457f0 2024-03-29T10:50:43.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 >> v1:172.21.15.5:6790/0 conn(0x7fae4413f650 legacy=0x7fae440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:43.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 >> v1:172.21.15.177:6789/0 conn(0x7fae440a9d70 legacy=0x7fae4414d8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:50:43.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:43.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 1 monclient: found mon.noname-a 2024-03-29T10:50:43.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:43.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:50:43.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae44146340 con 0x7fae441457f0 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient: _finish_auth 0 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fae3c0019f0 con 0x7fae441457f0 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.582+0000 7fae3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae3bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae3bfff640 10 monclient: dump: 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:43.566 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae3c001bd0 con 0x7fae441457f0 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae3bfff640 1 -- 172.21.15.5:0/517351204 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fae3c001fe0 con 0x7fae441457f0 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae3bfff640 10 monclient: dump: 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:43.567 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:43.568 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:43.568 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:43.568 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:43.568 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:43.568 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:43.568 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:43.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae4b5e9640 5 monclient: authenticate success, global_id 4314 2024-03-29T10:50:43.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae4b5e9640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:43.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.583+0000 7fae4b5e9640 4 set_mon_vals no callback set 2024-03-29T10:50:43.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.584+0000 7fae4b5e9640 10 monclient: shutdown 2024-03-29T10:50:43.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.584+0000 7fae4b5e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:43.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.584+0000 7fae4b5e9640 1 -- 172.21.15.5:0/517351204 >> v1:172.21.15.5:6789/0 conn(0x7fae441457f0 legacy=0x7fae4413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:43.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.584+0000 7fae4b5e9640 1 -- 172.21.15.5:0/517351204 shutdown_connections 2024-03-29T10:50:43.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.584+0000 7fae4b5e9640 1 -- 172.21.15.5:0/517351204 >> 172.21.15.5:0/517351204 conn(0x7fae44066a00 msgr2=0x7fae44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:43.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.584+0000 7fae4b5e9640 1 -- 172.21.15.5:0/517351204 shutdown_connections 2024-03-29T10:50:43.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.584+0000 7fae4b5e9640 1 -- 172.21.15.5:0/517351204 wait complete. 2024-03-29T10:50:43.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.585+0000 7fae4b5e9640 10 monclient: build_initial_monmap 2024-03-29T10:50:43.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.585+0000 7fae4b5e9640 10 monclient: monmap: 2024-03-29T10:50:43.571 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.586+0000 7fae4b5e9640 1 librados: starting msgr at 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.586+0000 7fae4b5e9640 1 librados: starting objecter 2024-03-29T10:50:43.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.586+0000 7fae4b5e9640 1 Processor -- start 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.586+0000 7fae4b5e9640 1 -- start start 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.586+0000 7fae4b5e9640 1 librados: setting wanted keys 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.586+0000 7fae4b5e9640 1 librados: calling monclient init 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.586+0000 7fae4b5e9640 10 monclient: init 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.587+0000 7fae4b5e9640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.587+0000 7fae4b5e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.587+0000 7fae4b5e9640 10 monclient(hunting): picked mon.noname-c con 0x7fae440a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.587+0000 7fae49385640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fae440a9d70 0x7fae441df330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:56896/0 (socket says 172.21.15.5:56896) 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.587+0000 7fae49385640 1 -- 172.21.15.5:0/3501161019 learned_addr learned my addr 172.21.15.5:0/3501161019 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.587+0000 7fae4b5e9640 10 monclient(hunting): picked mon.noname-a con 0x7fae4413f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.588+0000 7fae4b5e9640 10 monclient(hunting): picked mon.noname-b con 0x7fae441457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.588+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae440ab7c0 con 0x7fae4413f650 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.588+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae441dee10 con 0x7fae441457f0 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.588+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae440b1690 con 0x7fae440a9d70 2024-03-29T10:50:43.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.588+0000 7fae4b5e9640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:43.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.588+0000 7fae4b5e9640 10 monclient(hunting): authenticate will time out at 1056.829398s 2024-03-29T10:50:43.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.589+0000 7fae39ffb640 10 client.?.objecter ms_handle_connect 0x7fae440a9d70 2024-03-29T10:50:43.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.589+0000 7fae39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:43.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.589+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296418003 0 0) 0x7fae440b1690 con 0x7fae440a9d70 2024-03-29T10:50:43.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.589+0000 7fae39ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:43.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.589+0000 7fae39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:43.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.589+0000 7fae39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:43.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.589+0000 7fae39ffb640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:50:43.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.589+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae28004eb0 con 0x7fae440a9d70 2024-03-29T10:50:43.575 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.592+0000 7fae39ffb640 10 client.?.objecter ms_handle_connect 0x7fae441457f0 2024-03-29T10:50:43.575 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.592+0000 7fae39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:43.575 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.592+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86457815 0 0) 0x7fae28004eb0 con 0x7fae440a9d70 2024-03-29T10:50:43.575 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.592+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae440b1690 con 0x7fae440a9d70 2024-03-29T10:50:43.575 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.592+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fae3c001fe0 con 0x7fae440a9d70 2024-03-29T10:50:43.575 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.592+0000 7fae39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:43.575 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.592+0000 7fae39ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:43.575 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.592+0000 7fae39ffb640 10 monclient(hunting): dump: 2024-03-29T10:50:43.575 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:43.575 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.592+0000 7fae39ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.593+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1128229016 0 0) 0x7fae440b1690 con 0x7fae440a9d70 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.593+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 >> v1:172.21.15.177:6789/0 conn(0x7fae441457f0 legacy=0x7fae441e6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.593+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 >> v1:172.21.15.5:6789/0 conn(0x7fae4413f650 legacy=0x7fae440affc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:43.576 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.593+0000 7fae39ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:43.577 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae39ffb640 1 monclient: found mon.c 2024-03-29T10:50:43.577 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:43.577 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae39ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:43.577 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae441deaa0 con 0x7fae440a9d70 2024-03-29T10:50:43.577 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae39ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:43.577 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae39ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:43.578 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:43.578 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae4b5e9640 5 monclient: authenticate success, global_id 4361 2024-03-29T10:50:43.578 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae4b5e9640 10 monclient: _renew_subs 2024-03-29T10:50:43.578 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae4b5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:43.578 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.594+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae440b1120 con 0x7fae440a9d70 2024-03-29T10:50:43.578 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.595+0000 7fae4b5e9640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:43.578 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.595+0000 7fae4b5e9640 10 monclient: _renew_subs 2024-03-29T10:50:43.578 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.595+0000 7fae4b5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:43.578 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.595+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae440b08b0 con 0x7fae440a9d70 2024-03-29T10:50:43.578 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.595+0000 7fae4b5e9640 1 librados: init done 2024-03-29T10:50:43.579 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.596+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae3c001550 con 0x7fae440a9d70 2024-03-29T10:50:43.579 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.596+0000 7fae39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:43.579 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.596+0000 7fae3b7fe640 4 set_mon_vals no callback set 2024-03-29T10:50:43.579 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.597+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fae3c003f40 con 0x7fae440a9d70 2024-03-29T10:50:43.579 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.597+0000 7fae39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:43.579 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.597+0000 7fae39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.597+0000 7fae39ffb640 10 monclient: dump: 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.597+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fae3c004160 con 0x7fae440a9d70 2024-03-29T10:50:43.580 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.597+0000 7fae4b5e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:43.581 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.597+0000 7fae4b5e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:43.581 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.597+0000 7fae4b5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:43.581 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.597+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae24005080 con 0x7fae440a9d70 2024-03-29T10:50:43.582 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.598+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fae3c084fb0 con 0x7fae440a9d70 2024-03-29T10:50:43.582 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.598+0000 7fae48b84640 10 monclient: get_auth_request con 0x7fae2806dcf0 auth_method 0 2024-03-29T10:50:43.582 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.598+0000 7fae39ffb640 10 client.4361.objecter ms_dispatch 0x7fae44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:43.582 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.598+0000 7fae39ffb640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:43.582 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.598+0000 7fae39ffb640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:43.582 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.599+0000 7fae39ffb640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-29T10:50:43.582 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.599+0000 7fae39ffb640 10 client.4361.objecter ms_handle_connect 0x7fae2806dcf0 2024-03-29T10:50:43.588 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.606+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae3c050950 con 0x7fae440a9d70 2024-03-29T10:50:43.588 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.606+0000 7fae39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:43.588 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.606+0000 7fae39ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:43.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.936+0000 7fae4b5e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:43.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.936+0000 7fae4b5e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:43.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.936+0000 7fae4b5e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:43.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.936+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae24005330 con 0x7fae440a9d70 2024-03-29T10:50:43.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.940+0000 7fae39ffb640 1 -- 172.21.15.5:0/3501161019 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fae3c052880 con 0x7fae440a9d70 2024-03-29T10:50:43.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.940+0000 7fae39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:43.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.940+0000 7fae39ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:43.922 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:50:43.922 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.944+0000 7fae4b5e9640 10 librados: watch_flush enter 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.944+0000 7fae4b5e9640 10 librados: watch_flush exit 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.944+0000 7fae4b5e9640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.944+0000 7fae4b5e9640 10 client.4361.objecter successfully canceled tick 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.944+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 >> v1:172.21.15.5:6816/2847500151 conn(0x7fae2806dcf0 legacy=0x7fae280701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.944+0000 7fae4b5e9640 10 monclient: shutdown 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.944+0000 7fae4b5e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.944+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 >> v1:172.21.15.5:6790/0 conn(0x7fae440a9d70 legacy=0x7fae441df330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.945+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 shutdown_connections 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.945+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 >> 172.21.15.5:0/3501161019 conn(0x7fae44066a00 msgr2=0x7fae4413c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.945+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 shutdown_connections 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.945+0000 7fae4b5e9640 1 -- 172.21.15.5:0/3501161019 wait complete. 2024-03-29T10:50:43.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:43.945+0000 7fae4b5e9640 1 librados: shutdown 2024-03-29T10:50:43.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:50.751 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:50.904 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.921+0000 7f17e4ed3640 10 monclient: get_monmap_and_config 2024-03-29T10:50:50.904 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.921+0000 7f17e4ed3640 10 monclient: build_initial_monmap 2024-03-29T10:50:50.904 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.921+0000 7f17e4ed3640 10 monclient: monmap: 2024-03-29T10:50:50.904 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:50.904 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:50.904 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:50:50.922343+0000 2024-03-29T10:50:50.904 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:50:50.922343+0000 2024-03-29T10:50:50.904 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:50.904 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:50.905 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:50.905 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:50:50.905 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:50:50.905 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:50.905 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.923+0000 7f17e4ed3640 1 Processor -- start 2024-03-29T10:50:50.905 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.923+0000 7f17e4ed3640 1 -- start start 2024-03-29T10:50:50.905 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.923+0000 7f17e4ed3640 10 monclient: init 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17e4ed3640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17e4ed3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17e4ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f17e00b4100 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17e4ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f17e013ee90 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17e4ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f17e00a9ac0 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17e4ed3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e00af6f0 con 0x7f17e00a9ac0 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17e4ed3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e00aac80 con 0x7f17e013ee90 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17e4ed3640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e00aaf60 con 0x7f17e00b4100 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17e4ed3640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17e4ed3640 10 monclient(hunting): authenticate will time out at 1064.166331s 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17de59c640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f17e00b4100 0x7f17e00adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:46528/0 (socket says 172.21.15.5:46528) 2024-03-29T10:50:50.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.925+0000 7f17de59c640 1 -- 172.21.15.5:0/3701406503 learned_addr learned my addr 172.21.15.5:0/3701406503 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:50.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.927+0000 7f17dd59a640 1 -- 172.21.15.5:0/3701406503 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136473762 0 0) 0x7f17e00aaf60 con 0x7f17e00b4100 2024-03-29T10:50:50.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.927+0000 7f17dd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:50:50.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.927+0000 7f17dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:50.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.927+0000 7f17dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:50.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.927+0000 7f17dd59a640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:50:50.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.927+0000 7f17dd59a640 1 -- 172.21.15.5:0/3701406503 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c8003520 con 0x7f17e00b4100 2024-03-29T10:50:50.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.927+0000 7f17dd59a640 1 -- 172.21.15.5:0/3701406503 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 725436452 0 0) 0x7f17c8003520 con 0x7f17e00b4100 2024-03-29T10:50:50.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.928+0000 7f17dd59a640 1 -- 172.21.15.5:0/3701406503 >> v1:172.21.15.177:6789/0 conn(0x7f17e013ee90 legacy=0x7f17e013b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.928+0000 7f17dd59a640 1 -- 172.21.15.5:0/3701406503 >> v1:172.21.15.5:6789/0 conn(0x7f17e00a9ac0 legacy=0x7f17e0142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.928+0000 7f17dd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.928+0000 7f17dd59a640 1 monclient: found mon.noname-b 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 1 -- 172.21.15.5:0/3701406503 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e013e970 con 0x7f17e00b4100 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 10 monclient: _finish_auth 0 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17e4ed3640 5 monclient: authenticate success, global_id 4364 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17e4ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 1 -- 172.21.15.5:0/3701406503 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f17d4001690 con 0x7f17e00b4100 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:50.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 10 monclient: dump: 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17e4ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 1 -- 172.21.15.5:0/3701406503 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17d4001920 con 0x7f17e00b4100 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17e4ed3640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:50.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17e4ed3640 4 set_mon_vals no callback set 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 1 -- 172.21.15.5:0/3701406503 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f17d4001cb0 con 0x7f17e00b4100 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.929+0000 7f17dd59a640 10 monclient: dump: 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:50.916 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:50.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.930+0000 7f17e4ed3640 10 monclient: shutdown 2024-03-29T10:50:50.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.930+0000 7f17e4ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:50.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.930+0000 7f17e4ed3640 1 -- 172.21.15.5:0/3701406503 >> v1:172.21.15.5:6790/0 conn(0x7f17e00b4100 legacy=0x7f17e00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:50.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.930+0000 7f17e4ed3640 1 -- 172.21.15.5:0/3701406503 shutdown_connections 2024-03-29T10:50:50.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.930+0000 7f17e4ed3640 1 -- 172.21.15.5:0/3701406503 >> 172.21.15.5:0/3701406503 conn(0x7f17e00b35e0 msgr2=0x7f17e00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:50.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.931+0000 7f17e4ed3640 1 -- 172.21.15.5:0/3701406503 shutdown_connections 2024-03-29T10:50:50.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.931+0000 7f17e4ed3640 1 -- 172.21.15.5:0/3701406503 wait complete. 2024-03-29T10:50:50.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.932+0000 7f17e4ed3640 10 monclient: build_initial_monmap 2024-03-29T10:50:50.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.932+0000 7f17e4ed3640 10 monclient: monmap: 2024-03-29T10:50:50.918 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:50.918 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:50.918 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:50:50.918 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:50:50.918 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:50.918 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.933+0000 7f17e4ed3640 1 librados: starting msgr at 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.933+0000 7f17e4ed3640 1 librados: starting objecter 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.933+0000 7f17e4ed3640 1 Processor -- start 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.933+0000 7f17e4ed3640 1 -- start start 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.933+0000 7f17e4ed3640 1 librados: setting wanted keys 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.933+0000 7f17e4ed3640 1 librados: calling monclient init 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.933+0000 7f17e4ed3640 10 monclient: init 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.934+0000 7f17e4ed3640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.934+0000 7f17e4ed3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.934+0000 7f17e4ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f17e00a9ac0 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:50.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.934+0000 7f17e4ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f17e00b4100 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:50.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.934+0000 7f17e4ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f17e013ee90 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:50.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.934+0000 7f17e4ed3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e006eb90 con 0x7f17e00a9ac0 2024-03-29T10:50:50.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.934+0000 7f17e4ed3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e006ee00 con 0x7f17e00b4100 2024-03-29T10:50:50.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.934+0000 7f17e4ed3640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e006f0e0 con 0x7f17e013ee90 2024-03-29T10:50:50.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.934+0000 7f17e4ed3640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:50.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.934+0000 7f17e4ed3640 10 monclient(hunting): authenticate will time out at 1064.175232s 2024-03-29T10:50:50.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.936+0000 7f17ddd9b640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f17e00b4100 0x7f17e012b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:58363/0 (socket says 172.21.15.5:58363) 2024-03-29T10:50:50.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.936+0000 7f17ddd9b640 1 -- 172.21.15.5:0/887721179 learned_addr learned my addr 172.21.15.5:0/887721179 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:50.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.936+0000 7f17ceffd640 10 client.?.objecter ms_handle_connect 0x7f17e013ee90 2024-03-29T10:50:50.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.936+0000 7f17ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:50.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.936+0000 7f17ceffd640 10 client.?.objecter ms_handle_connect 0x7f17e00b4100 2024-03-29T10:50:50.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:50.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281364086 0 0) 0x7f17e006f0e0 con 0x7f17e013ee90 2024-03-29T10:50:50.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:50.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:50.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:50.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c0004eb0 con 0x7f17e013ee90 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555238824 0 0) 0x7f17e006ee00 con 0x7f17e00b4100 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e006f0e0 con 0x7f17e00b4100 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2541059178 0 0) 0x7f17c0004eb0 con 0x7f17e013ee90 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17e006ee00 con 0x7f17e013ee90 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f17d8001b70 con 0x7f17e013ee90 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.937+0000 7f17ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 10 monclient(hunting): dump: 2024-03-29T10:50:50.922 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 10 client.?.objecter ms_handle_connect 0x7f17e00a9ac0 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1587015657 0 0) 0x7f17e006ee00 con 0x7f17e013ee90 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 >> v1:172.21.15.177:6789/0 conn(0x7f17e00b4100 legacy=0x7f17e012b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:50.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 >> v1:172.21.15.5:6789/0 conn(0x7f17e00a9ac0 legacy=0x7f17e012aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 1 monclient: found mon.c 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e012a4f0 con 0x7f17e013ee90 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17e4ed3640 5 monclient: authenticate success, global_id 4367 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17e4ed3640 10 monclient: _renew_subs 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17e4ed3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17e4ed3640 1 -- 172.21.15.5:0/887721179 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e0142190 con 0x7f17e013ee90 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17e4ed3640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:50.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17e4ed3640 10 monclient: _renew_subs 2024-03-29T10:50:50.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17e4ed3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:50.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17e4ed3640 1 -- 172.21.15.5:0/887721179 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e006f500 con 0x7f17e013ee90 2024-03-29T10:50:50.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.938+0000 7f17e4ed3640 1 librados: init done 2024-03-29T10:50:50.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.940+0000 7f17e4ed3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:50.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.940+0000 7f17e4ed3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.940+0000 7f17e4ed3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.940+0000 7f17e4ed3640 1 -- 172.21.15.5:0/887721179 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e0075f10 con 0x7f17e013ee90 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.942+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17d8002080 con 0x7f17e013ee90 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.942+0000 7f17ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.942+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f17d8003b10 con 0x7f17e013ee90 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.942+0000 7f17ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.942+0000 7f17ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.942+0000 7f17ceffd640 10 monclient: dump: 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:50.926 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:50.927 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:50.927 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:50.927 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:50.927 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:50.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.942+0000 7f17dcd99640 4 set_mon_vals no callback set 2024-03-29T10:50:50.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.943+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f17d8019b60 con 0x7f17e013ee90 2024-03-29T10:50:50.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.944+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f17d8083a80 con 0x7f17e013ee90 2024-03-29T10:50:50.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.944+0000 7f17ceffd640 10 client.4367.objecter ms_dispatch 0x7f17e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:50.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.944+0000 7f17ceffd640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:50.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.944+0000 7f17ceffd640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:50.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.944+0000 7f17ceffd640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-29T10:50:50.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.944+0000 7f17de59c640 10 monclient: get_auth_request con 0x7f17c006dbc0 auth_method 0 2024-03-29T10:50:50.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.944+0000 7f17ceffd640 10 client.4367.objecter ms_handle_connect 0x7f17c006dbc0 2024-03-29T10:50:50.933 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.951+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17d804f190 con 0x7f17e013ee90 2024-03-29T10:50:50.933 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.951+0000 7f17ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:50.933 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:50.951+0000 7f17ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:51.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.277+0000 7f17e4ed3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:51.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.277+0000 7f17e4ed3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:51.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.277+0000 7f17e4ed3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:50:51.260 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.277+0000 7f17e4ed3640 1 -- 172.21.15.5:0/887721179 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17e012eaf0 con 0x7f17e013ee90 2024-03-29T10:50:51.260 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.278+0000 7f17ceffd640 1 -- 172.21.15.5:0/887721179 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f17d8051290 con 0x7f17e013ee90 2024-03-29T10:50:51.260 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.278+0000 7f17ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:51.260 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.278+0000 7f17ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:51.261 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:50:51.261 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:51.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.282+0000 7f17e4ed3640 10 librados: watch_flush enter 2024-03-29T10:50:51.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.282+0000 7f17e4ed3640 10 librados: watch_flush exit 2024-03-29T10:50:51.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.282+0000 7f17e4ed3640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-29T10:50:51.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.282+0000 7f17e4ed3640 10 client.4367.objecter successfully canceled tick 2024-03-29T10:50:51.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.282+0000 7f17e4ed3640 1 -- 172.21.15.5:0/887721179 >> v1:172.21.15.5:6816/2847500151 conn(0x7f17c006dbc0 legacy=0x7f17c0070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:51.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.282+0000 7f17e4ed3640 10 monclient: shutdown 2024-03-29T10:50:51.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.282+0000 7f17e4ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:51.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.282+0000 7f17e4ed3640 1 -- 172.21.15.5:0/887721179 >> v1:172.21.15.5:6790/0 conn(0x7f17e013ee90 legacy=0x7f17e01e6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:51.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.283+0000 7f17e4ed3640 1 -- 172.21.15.5:0/887721179 shutdown_connections 2024-03-29T10:50:51.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.283+0000 7f17e4ed3640 1 -- 172.21.15.5:0/887721179 >> 172.21.15.5:0/887721179 conn(0x7f17e00b35e0 msgr2=0x7f17e00b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.284+0000 7f17e4ed3640 1 -- 172.21.15.5:0/887721179 shutdown_connections 2024-03-29T10:50:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.284+0000 7f17e4ed3640 1 -- 172.21.15.5:0/887721179 wait complete. 2024-03-29T10:50:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:51.284+0000 7f17e4ed3640 1 librados: shutdown 2024-03-29T10:50:51.288 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:58.096 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.267+0000 7f672d0c5640 10 monclient: get_monmap_and_config 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.267+0000 7f672d0c5640 10 monclient: build_initial_monmap 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.267+0000 7f672d0c5640 10 monclient: monmap: 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:50:58.268925+0000 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:50:58.268925+0000 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:50:58.251 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:50:58.252 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:58.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.269+0000 7f672d0c5640 1 Processor -- start 2024-03-29T10:50:58.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.269+0000 7f672d0c5640 1 -- start start 2024-03-29T10:50:58.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.269+0000 7f672d0c5640 10 monclient: init 2024-03-29T10:50:58.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.270+0000 7f672d0c5640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:58.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.270+0000 7f672d0c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:50:58.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.270+0000 7f672d0c5640 10 monclient(hunting): picked mon.noname-a con 0x7f6720029eb0 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:58.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.271+0000 7f672d0c5640 10 monclient(hunting): picked mon.noname-b con 0x7f672002b0a0 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:58.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.271+0000 7f672d0c5640 10 monclient(hunting): picked mon.noname-c con 0x7f67200c9570 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:58.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.271+0000 7f672d0c5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672002a290 con 0x7f6720029eb0 2024-03-29T10:50:58.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.271+0000 7f672d0c5640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67200331c0 con 0x7f67200c9570 2024-03-29T10:50:58.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.271+0000 7f672d0c5640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672002c100 con 0x7f672002b0a0 2024-03-29T10:50:58.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.271+0000 7f672d0c5640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:58.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.271+0000 7f672d0c5640 10 monclient(hunting): authenticate will time out at 1071.512225s 2024-03-29T10:50:58.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.274+0000 7f672c8c4640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f67200c9570 0x7f67200d74c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:38583/0 (socket says 172.21.15.5:38583) 2024-03-29T10:50:58.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.274+0000 7f672c8c4640 1 -- 172.21.15.5:0/1594485646 learned_addr learned my addr 172.21.15.5:0/1594485646 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:58.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138384882 0 0) 0x7f67200331c0 con 0x7f67200c9570 2024-03-29T10:50:58.263 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:58.263 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6718003520 con 0x7f67200c9570 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283239141 0 0) 0x7f672002c100 con 0x7f672002b0a0 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.278+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67200331c0 con 0x7f672002b0a0 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842093396 0 0) 0x7f672002a290 con 0x7f6720029eb0 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:58.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6718004a30 con 0x7f6720029eb0 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 397571408 0 0) 0x7f67200331c0 con 0x7f672002b0a0 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 >> v1:172.21.15.177:6789/0 conn(0x7f67200c9570 legacy=0x7f67200d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 >> v1:172.21.15.5:6789/0 conn(0x7f6720029eb0 legacy=0x7f67200cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 1 monclient: found mon.noname-b 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:50:58.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672002a440 con 0x7f672002b0a0 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6714001670 con 0x7f672002b0a0 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient: dump: 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:58.266 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f672d0c5640 5 monclient: authenticate success, global_id 4370 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f672d0c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67140017d0 con 0x7f672002b0a0 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f6726ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f672d0c5640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:58.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.279+0000 7f672d0c5640 4 set_mon_vals no callback set 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.280+0000 7f6726ffd640 1 -- 172.21.15.5:0/1594485646 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6714001be0 con 0x7f672002b0a0 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.280+0000 7f6726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.280+0000 7f6726ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.280+0000 7f6726ffd640 10 monclient: dump: 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:58.268 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:58.269 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:58.269 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:58.269 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:58.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.280+0000 7f672d0c5640 10 monclient: shutdown 2024-03-29T10:50:58.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.280+0000 7f672d0c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:58.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.281+0000 7f672d0c5640 1 -- 172.21.15.5:0/1594485646 >> v1:172.21.15.5:6790/0 conn(0x7f672002b0a0 legacy=0x7f67200c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:58.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.283+0000 7f672d0c5640 1 -- 172.21.15.5:0/1594485646 shutdown_connections 2024-03-29T10:50:58.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.283+0000 7f672d0c5640 1 -- 172.21.15.5:0/1594485646 >> 172.21.15.5:0/1594485646 conn(0x7f67200058f0 msgr2=0x7f6720005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:58.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.285+0000 7f672d0c5640 1 -- 172.21.15.5:0/1594485646 shutdown_connections 2024-03-29T10:50:58.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.285+0000 7f672d0c5640 1 -- 172.21.15.5:0/1594485646 wait complete. 2024-03-29T10:50:58.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.286+0000 7f672d0c5640 10 monclient: build_initial_monmap 2024-03-29T10:50:58.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.286+0000 7f672d0c5640 10 monclient: monmap: 2024-03-29T10:50:58.271 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.287+0000 7f672d0c5640 1 librados: starting msgr at 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.287+0000 7f672d0c5640 1 librados: starting objecter 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.287+0000 7f672d0c5640 1 Processor -- start 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.287+0000 7f672d0c5640 1 -- start start 2024-03-29T10:50:58.272 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.287+0000 7f672d0c5640 1 librados: setting wanted keys 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.287+0000 7f672d0c5640 1 librados: calling monclient init 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.287+0000 7f672d0c5640 10 monclient: init 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.288+0000 7f672d0c5640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.288+0000 7f672d0c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.288+0000 7f672d0c5640 10 monclient(hunting): picked mon.noname-b con 0x7f6720029eb0 addr v1:172.21.15.177:6789/0 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.288+0000 7f672d0c5640 10 monclient(hunting): picked mon.noname-a con 0x7f672002b0a0 addr v1:172.21.15.5:6789/0 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.288+0000 7f672d0c5640 10 monclient(hunting): picked mon.noname-c con 0x7f67200c9570 addr v1:172.21.15.5:6790/0 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.288+0000 7f672d0c5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6720031430 con 0x7f672002b0a0 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.288+0000 7f672d0c5640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6720031ac0 con 0x7f6720029eb0 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.288+0000 7f672d0c5640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6720030630 con 0x7f67200c9570 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.288+0000 7f672d0c5640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:58.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.288+0000 7f672d0c5640 10 monclient(hunting): authenticate will time out at 1071.529907s 2024-03-29T10:50:58.275 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.294+0000 7f672c8c4640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f67200c9570 0x7f672016b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:53003/0 (socket says 172.21.15.5:53003) 2024-03-29T10:50:58.275 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.294+0000 7f672c8c4640 1 -- 172.21.15.5:0/1557210126 learned_addr learned my addr 172.21.15.5:0/1557210126 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:50:58.276 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.294+0000 7f6724ff9640 10 client.?.objecter ms_handle_connect 0x7f6720029eb0 2024-03-29T10:50:58.276 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:58.276 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057830514 0 0) 0x7f6720031ac0 con 0x7f6720029eb0 2024-03-29T10:50:58.276 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:50:58.276 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:58.277 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:58.277 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:50:58.277 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f670c004eb0 con 0x7f6720029eb0 2024-03-29T10:50:58.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 client.?.objecter ms_handle_connect 0x7f67200c9570 2024-03-29T10:50:58.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:58.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 client.?.objecter ms_handle_connect 0x7f672002b0a0 2024-03-29T10:50:58.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:58.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604566590 0 0) 0x7f6720031430 con 0x7f672002b0a0 2024-03-29T10:50:58.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:50:58.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.295+0000 7f6724ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:58.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:58.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6720031ac0 con 0x7f672002b0a0 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43907056 0 0) 0x7f6720030630 con 0x7f67200c9570 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6720031430 con 0x7f67200c9570 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3828548263 0 0) 0x7f6720031ac0 con 0x7f672002b0a0 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6720030630 con 0x7f672002b0a0 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6714001ae0 con 0x7f672002b0a0 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 10 monclient(hunting): dump: 2024-03-29T10:50:58.282 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.296+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 254749030 0 0) 0x7f6720031430 con 0x7f67200c9570 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6720031ac0 con 0x7f67200c9570 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f672804fdd0 con 0x7f67200c9570 2024-03-29T10:50:58.283 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 10 monclient(hunting): dump: 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4033172822 0 0) 0x7f6720030630 con 0x7f672002b0a0 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 >> v1:172.21.15.5:6790/0 conn(0x7f67200c9570 legacy=0x7f672016b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:58.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 >> v1:172.21.15.177:6789/0 conn(0x7f6720029eb0 legacy=0x7f672002f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 1 monclient: found mon.a 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67200316a0 con 0x7f672002b0a0 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 10 monclient: _finish_auth 0 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f6724ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f672d0c5640 5 monclient: authenticate success, global_id 4323 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f672d0c5640 10 monclient: _renew_subs 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f672d0c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f672d0c5640 1 -- 172.21.15.5:0/1557210126 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6720032690 con 0x7f672002b0a0 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.297+0000 7f672d0c5640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.298+0000 7f672d0c5640 10 monclient: _renew_subs 2024-03-29T10:50:58.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.298+0000 7f672d0c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.298+0000 7f672d0c5640 1 -- 172.21.15.5:0/1557210126 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6720030230 con 0x7f672002b0a0 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.298+0000 7f672d0c5640 1 librados: init done 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.298+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67140038a0 con 0x7f672002b0a0 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.298+0000 7f6724ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.298+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6714003bb0 con 0x7f672002b0a0 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.298+0000 7f6724ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.298+0000 7f6724ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.299+0000 7f67267fc640 4 set_mon_vals no callback set 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.299+0000 7f6724ff9640 10 monclient: dump: 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:50:58.286 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:58.287 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:50:58.287 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:50:58.287 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:50:58.287 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:50:58.287 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:50:58.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.300+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f6714019b80 con 0x7f672002b0a0 2024-03-29T10:50:58.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.301+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f6714083940 con 0x7f672002b0a0 2024-03-29T10:50:58.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.301+0000 7f6724ff9640 10 client.4323.objecter ms_dispatch 0x7f6728000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:58.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.301+0000 7f6724ff9640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:58.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.301+0000 7f6724ff9640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:58.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.301+0000 7f6724ff9640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-29T10:50:58.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.301+0000 7f6727fff640 10 monclient: get_auth_request con 0x7f670c06de40 auth_method 0 2024-03-29T10:50:58.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.301+0000 7f672d0c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:58.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.301+0000 7f672d0c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:58.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.301+0000 7f672d0c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:58.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.301+0000 7f672d0c5640 1 -- 172.21.15.5:0/1557210126 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6708005080 con 0x7f672002b0a0 2024-03-29T10:50:58.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.302+0000 7f6724ff9640 10 client.4323.objecter ms_handle_connect 0x7f670c06de40 2024-03-29T10:50:58.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.310+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f671404f220 con 0x7f672002b0a0 2024-03-29T10:50:58.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.310+0000 7f6724ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:58.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.310+0000 7f6724ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.640+0000 7f672d0c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.640+0000 7f672d0c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.640+0000 7f672d0c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:50:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.640+0000 7f672d0c5640 1 -- 172.21.15.5:0/1557210126 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6708005330 con 0x7f672002b0a0 2024-03-29T10:50:58.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.642+0000 7f6724ff9640 1 -- 172.21.15.5:0/1557210126 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f6714051150 con 0x7f672002b0a0 2024-03-29T10:50:58.624 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.642+0000 7f6724ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:58.624 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.642+0000 7f6724ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:58.624 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:50:58.624 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:58.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 10 librados: watch_flush enter 2024-03-29T10:50:58.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 10 librados: watch_flush exit 2024-03-29T10:50:58.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-29T10:50:58.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 10 client.4323.objecter successfully canceled tick 2024-03-29T10:50:58.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 1 -- 172.21.15.5:0/1557210126 >> v1:172.21.15.5:6816/2847500151 conn(0x7f670c06de40 legacy=0x7f670c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:58.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 10 monclient: shutdown 2024-03-29T10:50:58.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:58.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 1 -- 172.21.15.5:0/1557210126 >> v1:172.21.15.5:6789/0 conn(0x7f672002b0a0 legacy=0x7f6720031f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:58.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 1 -- 172.21.15.5:0/1557210126 shutdown_connections 2024-03-29T10:50:58.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 1 -- 172.21.15.5:0/1557210126 >> 172.21.15.5:0/1557210126 conn(0x7f67200058f0 msgr2=0x7f6720038480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:58.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.646+0000 7f672d0c5640 1 -- 172.21.15.5:0/1557210126 shutdown_connections 2024-03-29T10:50:58.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.647+0000 7f672d0c5640 1 -- 172.21.15.5:0/1557210126 wait complete. 2024-03-29T10:50:58.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:50:58.648+0000 7f672d0c5640 1 librados: shutdown 2024-03-29T10:50:58.649 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:05.456 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:05.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.622+0000 7f3030bba640 10 monclient: get_monmap_and_config 2024-03-29T10:51:05.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.622+0000 7f3030bba640 10 monclient: build_initial_monmap 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.622+0000 7f3030bba640 10 monclient: monmap: 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:51:05.623298+0000 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:51:05.623298+0000 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.623+0000 7f3030bba640 1 Processor -- start 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.623+0000 7f3030bba640 1 -- start start 2024-03-29T10:51:05.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.623+0000 7f3030bba640 10 monclient: init 2024-03-29T10:51:05.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.625+0000 7f3030bba640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:05.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.625+0000 7f3030bba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:05.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.625+0000 7f3030bba640 10 monclient(hunting): picked mon.noname-b con 0x7f302c0af6c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:05.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.625+0000 7f3030bba640 10 monclient(hunting): picked mon.noname-c con 0x7f302c14f6b0 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:05.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.625+0000 7f3030bba640 10 monclient(hunting): picked mon.noname-a con 0x7f302c152980 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:05.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.625+0000 7f3030bba640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0a94e0 con 0x7f302c152980 2024-03-29T10:51:05.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.625+0000 7f3030bba640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0afb60 con 0x7f302c14f6b0 2024-03-29T10:51:05.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.625+0000 7f3030bba640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0aab80 con 0x7f302c0af6c0 2024-03-29T10:51:05.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.625+0000 7f3030bba640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:05.611 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.625+0000 7f3030bba640 10 monclient(hunting): authenticate will time out at 1078.866907s 2024-03-29T10:51:05.611 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.628+0000 7f302ad9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f302c152980 0x7f302c152e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38884/0 (socket says 172.21.15.5:38884) 2024-03-29T10:51:05.611 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.628+0000 7f302ad9d640 1 -- 172.21.15.5:0/3793449394 learned_addr learned my addr 172.21.15.5:0/3793449394 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:05.611 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.629+0000 7f302959a640 1 -- 172.21.15.5:0/3793449394 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636477547 0 0) 0x7f302c0a94e0 con 0x7f302c152980 2024-03-29T10:51:05.611 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.629+0000 7f302959a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:05.611 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.629+0000 7f302959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:05.611 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.629+0000 7f302959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:05.611 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.629+0000 7f302959a640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:51:05.611 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.629+0000 7f302959a640 1 -- 172.21.15.5:0/3793449394 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3014003520 con 0x7f302c152980 2024-03-29T10:51:05.612 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.629+0000 7f302959a640 1 -- 172.21.15.5:0/3793449394 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3648768154 0 0) 0x7f3014003520 con 0x7f302c152980 2024-03-29T10:51:05.612 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.629+0000 7f302959a640 1 -- 172.21.15.5:0/3793449394 >> v1:172.21.15.5:6790/0 conn(0x7f302c0af6c0 legacy=0x7f302c0ad450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:05.613 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.632+0000 7f302959a640 1 -- 172.21.15.5:0/3793449394 >> v1:172.21.15.177:6789/0 conn(0x7f302c14f6b0 legacy=0x7f302c14fae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:05.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 10 monclient: _finish_hunting 0 2024-03-29T10:51:05.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 1 monclient: found mon.noname-a 2024-03-29T10:51:05.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:05.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:51:05.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 1 -- 172.21.15.5:0/3793449394 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c0aa680 con 0x7f302c152980 2024-03-29T10:51:05.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 10 monclient: _finish_auth 0 2024-03-29T10:51:05.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 10 monclient: _check_auth_tickets 2024-03-29T10:51:05.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:05.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 1 -- 172.21.15.5:0/3793449394 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f3024001a10 con 0x7f302c152980 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f302959a640 10 monclient: dump: 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f3030bba640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:51:05.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.633+0000 7f3030bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.634+0000 7f302959a640 1 -- 172.21.15.5:0/3793449394 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3024001c70 con 0x7f302c152980 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.634+0000 7f302959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.634+0000 7f3030bba640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.634+0000 7f3030bba640 4 set_mon_vals no callback set 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.634+0000 7f302959a640 1 -- 172.21.15.5:0/3793449394 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f30240020c0 con 0x7f302c152980 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.634+0000 7f302959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.634+0000 7f302959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.634+0000 7f302959a640 10 monclient: dump: 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:05.618 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:05.619 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:05.619 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:05.619 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:05.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.635+0000 7f3030bba640 10 monclient: shutdown 2024-03-29T10:51:05.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.635+0000 7f3030bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:05.620 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.635+0000 7f3030bba640 1 -- 172.21.15.5:0/3793449394 >> v1:172.21.15.5:6789/0 conn(0x7f302c152980 legacy=0x7f302c152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:05.620 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.635+0000 7f3030bba640 1 -- 172.21.15.5:0/3793449394 shutdown_connections 2024-03-29T10:51:05.620 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.635+0000 7f3030bba640 1 -- 172.21.15.5:0/3793449394 >> 172.21.15.5:0/3793449394 conn(0x7f302c0b35e0 msgr2=0x7f302c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:05.621 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.636+0000 7f3030bba640 1 -- 172.21.15.5:0/3793449394 shutdown_connections 2024-03-29T10:51:05.621 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.636+0000 7f3030bba640 1 -- 172.21.15.5:0/3793449394 wait complete. 2024-03-29T10:51:05.621 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.637+0000 7f3030bba640 10 monclient: build_initial_monmap 2024-03-29T10:51:05.621 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.637+0000 7f3030bba640 10 monclient: monmap: 2024-03-29T10:51:05.621 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:05.621 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:05.621 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:51:05.621 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:51:05.621 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.637+0000 7f3030bba640 1 librados: starting msgr at 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.637+0000 7f3030bba640 1 librados: starting objecter 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.637+0000 7f3030bba640 1 Processor -- start 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.637+0000 7f3030bba640 1 -- start start 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.637+0000 7f3030bba640 1 librados: setting wanted keys 2024-03-29T10:51:05.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.637+0000 7f3030bba640 1 librados: calling monclient init 2024-03-29T10:51:05.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.637+0000 7f3030bba640 10 monclient: init 2024-03-29T10:51:05.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.638+0000 7f3030bba640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:05.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.638+0000 7f3030bba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:05.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.638+0000 7f3030bba640 10 monclient(hunting): picked mon.noname-b con 0x7f302c0af6c0 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:05.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.638+0000 7f3030bba640 10 monclient(hunting): picked mon.noname-a con 0x7f302c14f6b0 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:05.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.638+0000 7f3030bba640 10 monclient(hunting): picked mon.noname-c con 0x7f302c152980 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:05.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.638+0000 7f3030bba640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c1482f0 con 0x7f302c14f6b0 2024-03-29T10:51:05.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.638+0000 7f3030bba640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c1485d0 con 0x7f302c0af6c0 2024-03-29T10:51:05.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.638+0000 7f3030bba640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0723c0 con 0x7f302c152980 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.638+0000 7f3030bba640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.638+0000 7f3030bba640 10 monclient(hunting): authenticate will time out at 1078.879639s 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.642+0000 7f3029d9b640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f302c14f6b0 0x7f302c070a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33890/0 (socket says 172.21.15.5:33890) 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.642+0000 7f3029d9b640 1 -- 172.21.15.5:0/3526403434 learned_addr learned my addr 172.21.15.5:0/3526403434 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 client.?.objecter ms_handle_connect 0x7f302c0af6c0 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 client.?.objecter ms_handle_connect 0x7f302c14f6b0 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255657678 0 0) 0x7f302c1485d0 con 0x7f302c0af6c0 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c004eb0 con 0x7f302c0af6c0 2024-03-29T10:51:05.628 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918135760 0 0) 0x7f302c1482f0 con 0x7f302c14f6b0 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.643+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c0051d0 con 0x7f302c14f6b0 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.644+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 553404010 0 0) 0x7f300c004eb0 con 0x7f302c0af6c0 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.644+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f300c005640 con 0x7f302c0af6c0 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.644+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f301c001830 con 0x7f302c0af6c0 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.644+0000 7f301affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.644+0000 7f301affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.644+0000 7f301affd640 10 monclient(hunting): dump: 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:05.629 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.644+0000 7f301affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.645+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894982554 0 0) 0x7f300c0051d0 con 0x7f302c14f6b0 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.645+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f300c005d30 con 0x7f302c14f6b0 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.645+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f30200017f0 con 0x7f302c14f6b0 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.645+0000 7f301affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.645+0000 7f301affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.645+0000 7f301affd640 10 monclient(hunting): dump: 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:05.630 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:05.631 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:05.631 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:05.631 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:05.631 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:05.631 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:05.631 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:05.631 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:05.631 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:05.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 402245070 0 0) 0x7f300c005640 con 0x7f302c0af6c0 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 >> v1:172.21.15.5:6790/0 conn(0x7f302c152980 legacy=0x7f302c1f2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 >> v1:172.21.15.5:6789/0 conn(0x7f302c14f6b0 legacy=0x7f302c070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 1 monclient: found mon.b 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c08a1e0 con 0x7f302c0af6c0 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 10 monclient: _finish_auth 0 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f301affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f3030bba640 5 monclient: authenticate success, global_id 4330 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f3030bba640 10 monclient: _renew_subs 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f3030bba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:51:05.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f3030bba640 1 -- 172.21.15.5:0/3526403434 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c088460 con 0x7f302c0af6c0 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f3030bba640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f3030bba640 10 monclient: _renew_subs 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f3030bba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f3030bba640 1 -- 172.21.15.5:0/3526403434 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c08b2f0 con 0x7f302c0af6c0 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.646+0000 7f3030bba640 1 librados: init done 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.647+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f301c001d40 con 0x7f302c0af6c0 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.647+0000 7f301affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.647+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f301c001830 con 0x7f302c0af6c0 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.647+0000 7f301affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.647+0000 7f301affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.647+0000 7f301affd640 10 monclient: dump: 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:05.636 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.648+0000 7f301bfff640 4 set_mon_vals no callback set 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.649+0000 7f3030bba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.649+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f301c0195a0 con 0x7f302c0af6c0 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.649+0000 7f3030bba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.649+0000 7f3030bba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.649+0000 7f3030bba640 1 -- 172.21.15.5:0/3526403434 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3010005080 con 0x7f302c0af6c0 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.650+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f301c083360 con 0x7f302c0af6c0 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.650+0000 7f301affd640 10 client.4330.objecter ms_dispatch 0x7f302c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:05.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.650+0000 7f301affd640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:05.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.650+0000 7f301affd640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:05.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.651+0000 7f301affd640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T10:51:05.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.651+0000 7f3029d9b640 10 monclient: get_auth_request con 0x7f300c06e960 auth_method 0 2024-03-29T10:51:05.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.651+0000 7f301affd640 10 client.4330.objecter ms_handle_connect 0x7f300c06e960 2024-03-29T10:51:05.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.658+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f301c04ec40 con 0x7f302c0af6c0 2024-03-29T10:51:05.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.658+0000 7f301affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:05.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.658+0000 7f301affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:05.968 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.986+0000 7f3030bba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:05.969 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.986+0000 7f3030bba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:05.969 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.986+0000 7f3030bba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:51:05.969 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.986+0000 7f3030bba640 1 -- 172.21.15.5:0/3526403434 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3010005330 con 0x7f302c0af6c0 2024-03-29T10:51:05.969 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.988+0000 7f301affd640 1 -- 172.21.15.5:0/3526403434 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f301c04ec40 con 0x7f302c0af6c0 2024-03-29T10:51:05.969 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.988+0000 7f301affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:05.969 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.988+0000 7f301affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:05.972 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:51:05.973 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.991+0000 7f3030bba640 10 librados: watch_flush enter 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.993+0000 7f3030bba640 10 librados: watch_flush exit 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.993+0000 7f3030bba640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.993+0000 7f3030bba640 10 client.4330.objecter successfully canceled tick 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.994+0000 7f3030bba640 1 -- 172.21.15.5:0/3526403434 >> v1:172.21.15.5:6816/2847500151 conn(0x7f300c06e960 legacy=0x7f300c070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.994+0000 7f3030bba640 10 monclient: shutdown 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.994+0000 7f3030bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.994+0000 7f3030bba640 1 -- 172.21.15.5:0/3526403434 >> v1:172.21.15.177:6789/0 conn(0x7f302c0af6c0 legacy=0x7f302c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.994+0000 7f3030bba640 1 -- 172.21.15.5:0/3526403434 shutdown_connections 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.994+0000 7f3030bba640 1 -- 172.21.15.5:0/3526403434 >> 172.21.15.5:0/3526403434 conn(0x7f302c0b35e0 msgr2=0x7f302c0ac0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.994+0000 7f3030bba640 1 -- 172.21.15.5:0/3526403434 shutdown_connections 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.994+0000 7f3030bba640 1 -- 172.21.15.5:0/3526403434 wait complete. 2024-03-29T10:51:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:05.994+0000 7f3030bba640 1 librados: shutdown 2024-03-29T10:51:05.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:12.801 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:12.949 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.967+0000 7f9a13c27640 10 monclient: get_monmap_and_config 2024-03-29T10:51:12.949 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.967+0000 7f9a13c27640 10 monclient: build_initial_monmap 2024-03-29T10:51:12.949 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.967+0000 7f9a13c27640 10 monclient: monmap: 2024-03-29T10:51:12.949 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:12.949 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:12.949 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:51:12.968512+0000 2024-03-29T10:51:12.950 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:51:12.968512+0000 2024-03-29T10:51:12.950 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:12.950 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:12.950 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:12.950 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:51:12.950 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:51:12.950 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:12.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.968+0000 7f9a13c27640 1 Processor -- start 2024-03-29T10:51:12.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.968+0000 7f9a13c27640 1 -- start start 2024-03-29T10:51:12.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.968+0000 7f9a13c27640 10 monclient: init 2024-03-29T10:51:12.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.970+0000 7f9a13c27640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:12.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.970+0000 7f9a13c27640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:12.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.971+0000 7f9a119c3640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9a0c0b0770 0x7f9a0c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53922/0 (socket says 172.21.15.5:53922) 2024-03-29T10:51:12.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.971+0000 7f9a119c3640 1 -- 172.21.15.5:0/3972967345 learned_addr learned my addr 172.21.15.5:0/3972967345 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:12.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.971+0000 7f9a13c27640 10 monclient(hunting): picked mon.noname-a con 0x7f9a0c0b0770 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:12.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.971+0000 7f9a13c27640 10 monclient(hunting): picked mon.noname-c con 0x7f9a0c14a700 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:12.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.971+0000 7f9a13c27640 10 monclient(hunting): picked mon.noname-b con 0x7f9a0c0ac5a0 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:12.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.971+0000 7f9a13c27640 1 -- 172.21.15.5:0/3972967345 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c0aba70 con 0x7f9a0c0b0770 2024-03-29T10:51:12.953 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.971+0000 7f9a13c27640 1 -- 172.21.15.5:0/3972967345 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c0aecb0 con 0x7f9a0c14a700 2024-03-29T10:51:12.954 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.971+0000 7f9a13c27640 1 -- 172.21.15.5:0/3972967345 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c0af150 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.954 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.971+0000 7f9a13c27640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:12.954 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.971+0000 7f9a13c27640 10 monclient(hunting): authenticate will time out at 1086.212879s 2024-03-29T10:51:12.954 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482954426 0 0) 0x7f9a0c0af150 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:51:12.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f8003520 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175324524 0 0) 0x7f9a0c0aba70 con 0x7f9a0c0b0770 2024-03-29T10:51:12.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:51:12.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.973+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c0af150 con 0x7f9a0c0b0770 2024-03-29T10:51:12.957 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3490918999 0 0) 0x7f9a0c0af150 con 0x7f9a0c0b0770 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 >> v1:172.21.15.5:6790/0 conn(0x7f9a0c0ac5a0 legacy=0x7f9a0c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 >> v1:172.21.15.177:6789/0 conn(0x7f9a0c14a700 legacy=0x7f9a0c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: _finish_hunting 0 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 1 monclient: found mon.noname-a 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c0ac9d0 con 0x7f9a0c0b0770 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: _finish_auth 0 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: _check_auth_tickets 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9a080015f0 con 0x7f9a0c0b0770 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: dump: 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:12.959 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:12.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a08001750 con 0x7f9a0c0b0770 2024-03-29T10:51:12.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:12.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 1 -- 172.21.15.5:0/3972967345 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9a08001b60 con 0x7f9a0c0b0770 2024-03-29T10:51:12.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:12.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.976+0000 7f9a109c1640 10 monclient: dump: 2024-03-29T10:51:12.960 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:12.960 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.977+0000 7f9a13c27640 5 monclient: authenticate success, global_id 4332 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.977+0000 7f9a13c27640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:12.961 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.977+0000 7f9a13c27640 4 set_mon_vals no callback set 2024-03-29T10:51:12.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.980+0000 7f9a13c27640 10 monclient: shutdown 2024-03-29T10:51:12.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.980+0000 7f9a13c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:12.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.980+0000 7f9a13c27640 1 -- 172.21.15.5:0/3972967345 >> v1:172.21.15.5:6789/0 conn(0x7f9a0c0b0770 legacy=0x7f9a0c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.980+0000 7f9a13c27640 1 -- 172.21.15.5:0/3972967345 shutdown_connections 2024-03-29T10:51:12.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.980+0000 7f9a13c27640 1 -- 172.21.15.5:0/3972967345 >> 172.21.15.5:0/3972967345 conn(0x7f9a0c06b660 msgr2=0x7f9a0c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:12.963 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.982+0000 7f9a13c27640 1 -- 172.21.15.5:0/3972967345 shutdown_connections 2024-03-29T10:51:12.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.982+0000 7f9a13c27640 1 -- 172.21.15.5:0/3972967345 wait complete. 2024-03-29T10:51:12.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.983+0000 7f9a13c27640 10 monclient: build_initial_monmap 2024-03-29T10:51:12.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.983+0000 7f9a13c27640 10 monclient: monmap: 2024-03-29T10:51:12.964 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:12.965 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:12.965 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:51:12.965 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:51:12.965 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:12.965 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:12.965 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:12.965 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:51:12.965 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:51:12.965 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:12.966 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.984+0000 7f9a13c27640 1 librados: starting msgr at 2024-03-29T10:51:12.966 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.984+0000 7f9a13c27640 1 librados: starting objecter 2024-03-29T10:51:12.966 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.984+0000 7f9a13c27640 1 Processor -- start 2024-03-29T10:51:12.966 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.984+0000 7f9a13c27640 1 -- start start 2024-03-29T10:51:12.966 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.984+0000 7f9a13c27640 1 librados: setting wanted keys 2024-03-29T10:51:12.966 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.984+0000 7f9a13c27640 1 librados: calling monclient init 2024-03-29T10:51:12.967 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.984+0000 7f9a13c27640 10 monclient: init 2024-03-29T10:51:12.967 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.985+0000 7f9a13c27640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:12.967 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.985+0000 7f9a13c27640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:12.970 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.988+0000 7f9a13c27640 10 monclient(hunting): picked mon.noname-b con 0x7f9a0c0ac5a0 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:12.970 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.988+0000 7f9a13c27640 10 monclient(hunting): picked mon.noname-c con 0x7f9a0c0b0770 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:12.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.988+0000 7f9a13c27640 10 monclient(hunting): picked mon.noname-a con 0x7f9a0c14a700 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:12.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.989+0000 7f9a13c27640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c129190 con 0x7f9a0c14a700 2024-03-29T10:51:12.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.989+0000 7f9a13c27640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c129320 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.989+0000 7f9a13c27640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c147aa0 con 0x7f9a0c0b0770 2024-03-29T10:51:12.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.989+0000 7f9a13c27640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:12.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.989+0000 7f9a13c27640 10 monclient(hunting): authenticate will time out at 1086.230118s 2024-03-29T10:51:12.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.989+0000 7f9a119c3640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f9a0c0ac5a0 0x7f9a0c146d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:48855/0 (socket says 172.21.15.5:48855) 2024-03-29T10:51:12.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.989+0000 7f9a119c3640 1 -- 172.21.15.5:0/1646367158 learned_addr learned my addr 172.21.15.5:0/1646367158 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:12.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 client.?.objecter ms_handle_connect 0x7f9a0c0ac5a0 2024-03-29T10:51:12.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:12.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 client.?.objecter ms_handle_connect 0x7f9a0c14a700 2024-03-29T10:51:12.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:12.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 client.?.objecter ms_handle_connect 0x7f9a0c0b0770 2024-03-29T10:51:12.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:12.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931059370 0 0) 0x7f9a0c129320 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f0004eb0 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743653264 0 0) 0x7f9a0c147aa0 con 0x7f9a0c0b0770 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c129320 con 0x7f9a0c0b0770 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628252518 0 0) 0x7f9a0c129190 con 0x7f9a0c14a700 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:51:12.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.993+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c147aa0 con 0x7f9a0c14a700 2024-03-29T10:51:12.977 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.994+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2209873834 0 0) 0x7f99f0004eb0 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.977 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.994+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a0c129190 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.977 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.994+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9a08001b20 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.977 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.994+0000 7f9a027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.978 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.994+0000 7f9a027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:12.978 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.994+0000 7f9a027fc640 10 monclient(hunting): dump: 2024-03-29T10:51:12.978 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:12.978 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:12.978 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:12.978 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:12.978 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.978 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.994+0000 7f9a027fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.995+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 938297171 0 0) 0x7f9a0c129190 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.995+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 >> v1:172.21.15.5:6790/0 conn(0x7f9a0c0b0770 legacy=0x7f9a0c1db640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.995+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 >> v1:172.21.15.5:6789/0 conn(0x7f9a0c14a700 legacy=0x7f9a0c1e6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 1 monclient: found mon.b 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c147c30 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:12.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a13c27640 5 monclient: authenticate success, global_id 4336 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a13c27640 10 monclient: _renew_subs 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a13c27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a13c27640 1 -- 172.21.15.5:0/1646367158 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0c127c90 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a08004b00 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a13c27640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a13c27640 10 monclient: _renew_subs 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9a08004f10 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a13c27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a13c27640 1 -- 172.21.15.5:0/1646367158 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0c1286e0 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a13c27640 1 librados: init done 2024-03-29T10:51:12.980 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:12.981 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:12.981 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a027fc640 10 monclient: dump: 2024-03-29T10:51:12.981 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:12.981 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.996+0000 7f9a03fff640 4 set_mon_vals no callback set 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.998+0000 7f99effff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.998+0000 7f99effff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.998+0000 7f99effff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:51:12.982 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:12.998+0000 7f99effff640 1 -- 172.21.15.5:0/1646367158 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0c0ab560 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.983 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.001+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f9a0801aee0 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.983 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.002+0000 7f9a111c2640 10 monclient: get_auth_request con 0x7f99f006dce0 auth_method 0 2024-03-29T10:51:12.984 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.003+0000 7f9a027fc640 10 client.4336.objecter ms_handle_connect 0x7f99f006dce0 2024-03-29T10:51:12.984 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.003+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f9a08084b60 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.985 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.003+0000 7f9a027fc640 10 client.4336.objecter ms_dispatch 0x7f9a0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:12.985 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.003+0000 7f9a027fc640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:12.985 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.003+0000 7f9a027fc640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:12.985 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.003+0000 7f9a027fc640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T10:51:12.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.009+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a08050440 con 0x7f9a0c0ac5a0 2024-03-29T10:51:12.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.009+0000 7f9a027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:12.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.009+0000 7f9a027fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:13.364 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.382+0000 7f99effff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:13.364 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.382+0000 7f99effff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:13.365 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.382+0000 7f99effff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:51:13.365 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.382+0000 7f99effff640 1 -- 172.21.15.5:0/1646367158 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a0c128a30 con 0x7f9a0c0ac5a0 2024-03-29T10:51:13.365 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.384+0000 7f9a027fc640 1 -- 172.21.15.5:0/1646367158 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f9a08052370 con 0x7f9a0c0ac5a0 2024-03-29T10:51:13.365 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.384+0000 7f9a027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:13.365 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.384+0000 7f9a027fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:13.366 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:51:13.366 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:13.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.387+0000 7f99effff640 10 librados: watch_flush enter 2024-03-29T10:51:13.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.387+0000 7f99effff640 10 librados: watch_flush exit 2024-03-29T10:51:13.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.387+0000 7f99effff640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T10:51:13.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.388+0000 7f99effff640 10 client.4336.objecter successfully canceled tick 2024-03-29T10:51:13.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.388+0000 7f99effff640 1 -- 172.21.15.5:0/1646367158 >> v1:172.21.15.5:6816/2847500151 conn(0x7f99f006dce0 legacy=0x7f99f00701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:13.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.388+0000 7f99effff640 10 monclient: shutdown 2024-03-29T10:51:13.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.388+0000 7f99effff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:13.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.388+0000 7f99effff640 1 -- 172.21.15.5:0/1646367158 >> v1:172.21.15.177:6789/0 conn(0x7f9a0c0ac5a0 legacy=0x7f9a0c146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:13.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.388+0000 7f99effff640 1 -- 172.21.15.5:0/1646367158 shutdown_connections 2024-03-29T10:51:13.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.388+0000 7f99effff640 1 -- 172.21.15.5:0/1646367158 >> 172.21.15.5:0/1646367158 conn(0x7f9a0c06b660 msgr2=0x7f9a0c071450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:13.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.389+0000 7f99effff640 1 -- 172.21.15.5:0/1646367158 shutdown_connections 2024-03-29T10:51:13.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.389+0000 7f99effff640 1 -- 172.21.15.5:0/1646367158 wait complete. 2024-03-29T10:51:13.373 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:13.389+0000 7f99effff640 1 librados: shutdown 2024-03-29T10:51:13.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:20.195 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:20.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.367+0000 7f8106f64640 10 monclient: get_monmap_and_config 2024-03-29T10:51:20.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.367+0000 7f8106f64640 10 monclient: build_initial_monmap 2024-03-29T10:51:20.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.367+0000 7f8106f64640 10 monclient: monmap: 2024-03-29T10:51:20.349 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:20.349 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:20.349 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:51:20.368137+0000 2024-03-29T10:51:20.349 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:51:20.368137+0000 2024-03-29T10:51:20.349 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:20.349 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:20.349 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:20.349 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:51:20.349 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:51:20.349 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:20.350 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.367+0000 7f8106f64640 1 Processor -- start 2024-03-29T10:51:20.350 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.368+0000 7f8106f64640 1 -- start start 2024-03-29T10:51:20.350 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.368+0000 7f8106f64640 10 monclient: init 2024-03-29T10:51:20.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8106f64640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:20.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8106f64640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:20.351 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8106f64640 10 monclient(hunting): picked mon.noname-c con 0x7f8100145430 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:20.352 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8106f64640 10 monclient(hunting): picked mon.noname-a con 0x7f8100145810 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:20.352 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8106f64640 10 monclient(hunting): picked mon.noname-b con 0x7f81000b32b0 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:20.352 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8106f64640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000b26c0 con 0x7f8100145810 2024-03-29T10:51:20.352 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8106f64640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000aafd0 con 0x7f8100145430 2024-03-29T10:51:20.352 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8105761640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f8100145810 0x7f81000b2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35744/0 (socket says 172.21.15.5:35744) 2024-03-29T10:51:20.352 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8105761640 1 -- 172.21.15.5:0/3882959852 learned_addr learned my addr 172.21.15.5:0/3882959852 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:20.352 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8106f64640 1 -- 172.21.15.5:0/3882959852 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000b3970 con 0x7f81000b32b0 2024-03-29T10:51:20.352 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8106f64640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:20.352 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8106f64640 10 monclient(hunting): authenticate will time out at 1093.610815s 2024-03-29T10:51:20.352 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.369+0000 7f8105f62640 1 --1- 172.21.15.5:0/3882959852 >> v1:172.21.15.177:6789/0 conn(0x7f8100145430 0x7f81000b1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:33890/0 (socket says 172.21.15.5:33890) 2024-03-29T10:51:20.355 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445324851 0 0) 0x7f81000b26c0 con 0x7f8100145810 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80ec003520 con 0x7f8100145810 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188086566 0 0) 0x7f81000aafd0 con 0x7f8100145430 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81000b26c0 con 0x7f8100145430 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000848099 0 0) 0x7f81000b3970 con 0x7f81000b32b0 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.370+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81000aafd0 con 0x7f81000b32b0 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2392728432 0 0) 0x7f81000aafd0 con 0x7f81000b32b0 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 >> v1:172.21.15.177:6789/0 conn(0x7f8100145430 legacy=0x7f81000b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 >> v1:172.21.15.5:6789/0 conn(0x7f8100145810 legacy=0x7f81000b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 10 monclient: _finish_hunting 0 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 1 monclient: found mon.noname-b 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000b3b00 con 0x7f81000b32b0 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 10 monclient: _finish_auth 0 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 10 monclient: _check_auth_tickets 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:20.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8106f64640 5 monclient: authenticate success, global_id 4382 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8106f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f80f80019b0 con 0x7f81000b32b0 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 10 monclient: dump: 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:20.358 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8106f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80f8001b10 con 0x7f81000b32b0 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8104f60640 1 -- 172.21.15.5:0/3882959852 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f80f8001ee0 con 0x7f81000b32b0 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8106f64640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.371+0000 7f8106f64640 4 set_mon_vals no callback set 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.372+0000 7f8104f60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.372+0000 7f8104f60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.372+0000 7f8104f60640 10 monclient: dump: 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:20.359 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.372+0000 7f8106f64640 10 monclient: shutdown 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.372+0000 7f8106f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.372+0000 7f8106f64640 1 -- 172.21.15.5:0/3882959852 >> v1:172.21.15.5:6790/0 conn(0x7f81000b32b0 legacy=0x7f810014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.372+0000 7f8106f64640 1 -- 172.21.15.5:0/3882959852 shutdown_connections 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.372+0000 7f8106f64640 1 -- 172.21.15.5:0/3882959852 >> 172.21.15.5:0/3882959852 conn(0x7f810006d140 msgr2=0x7f8100066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.372+0000 7f8106f64640 1 -- 172.21.15.5:0/3882959852 shutdown_connections 2024-03-29T10:51:20.360 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.373+0000 7f8106f64640 1 -- 172.21.15.5:0/3882959852 wait complete. 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.374+0000 7f8106f64640 10 monclient: build_initial_monmap 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.374+0000 7f8106f64640 10 monclient: monmap: 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.374+0000 7f8106f64640 1 librados: starting msgr at 2024-03-29T10:51:20.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.374+0000 7f8106f64640 1 librados: starting objecter 2024-03-29T10:51:20.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.374+0000 7f8106f64640 1 Processor -- start 2024-03-29T10:51:20.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.374+0000 7f8106f64640 1 -- start start 2024-03-29T10:51:20.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.374+0000 7f8106f64640 1 librados: setting wanted keys 2024-03-29T10:51:20.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.374+0000 7f8106f64640 1 librados: calling monclient init 2024-03-29T10:51:20.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.374+0000 7f8106f64640 10 monclient: init 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106f64640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106f64640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106f64640 10 monclient(hunting): picked mon.noname-c con 0x7f81000b32b0 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106f64640 10 monclient(hunting): picked mon.noname-a con 0x7f8100145430 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106f64640 10 monclient(hunting): picked mon.noname-b con 0x7f810013e4f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:20.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106f64640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810013eaa0 con 0x7f8100145430 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106f64640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810013dc80 con 0x7f810013e4f0 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106763640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f810013e4f0 0x7f81001e26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:34318/0 (socket says 172.21.15.5:34318) 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106763640 1 -- 172.21.15.5:0/4190069983 learned_addr learned my addr 172.21.15.5:0/4190069983 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810013f0d0 con 0x7f81000b32b0 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106f64640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.375+0000 7f8106f64640 10 monclient(hunting): authenticate will time out at 1093.616952s 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 client.?.objecter ms_handle_connect 0x7f810013e4f0 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 client.?.objecter ms_handle_connect 0x7f8100145430 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 client.?.objecter ms_handle_connect 0x7f81000b32b0 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107945942 0 0) 0x7f810013f0d0 con 0x7f81000b32b0 2024-03-29T10:51:20.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e4004eb0 con 0x7f81000b32b0 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590378409 0 0) 0x7f810013eaa0 con 0x7f8100145430 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.376+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f810013f0d0 con 0x7f8100145430 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366993226 0 0) 0x7f810013dc80 con 0x7f810013e4f0 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:20.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): my global_id is 4342 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f810013eaa0 con 0x7f810013e4f0 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3923362708 0 0) 0x7f80e4004eb0 con 0x7f81000b32b0 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f810013dc80 con 0x7f81000b32b0 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f80fc0016b0 con 0x7f81000b32b0 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): dump: 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:20.371 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1048880029 0 0) 0x7f810013eaa0 con 0x7f810013e4f0 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80e4004eb0 con 0x7f810013e4f0 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f80f8001e20 con 0x7f810013e4f0 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): dump: 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.372 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.374 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.374 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:20.374 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:20.374 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:20.374 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:20.374 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:20.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2402686638 0 0) 0x7f810013f0d0 con 0x7f8100145430 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f810013eaa0 con 0x7f8100145430 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f80f0001710 con 0x7f8100145430 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.377+0000 7f80f67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 10 monclient(hunting): dump: 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:20.375 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1369020632 0 0) 0x7f810013dc80 con 0x7f81000b32b0 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 >> v1:172.21.15.177:6789/0 conn(0x7f810013e4f0 legacy=0x7f81001e26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 >> v1:172.21.15.5:6789/0 conn(0x7f8100145430 legacy=0x7f81001da350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 1 monclient: found mon.c 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8100142000 con 0x7f81000b32b0 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f80f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f8106f64640 5 monclient: authenticate success, global_id 4385 2024-03-29T10:51:20.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f8106f64640 10 monclient: _renew_subs 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f8106f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8100142250 con 0x7f81000b32b0 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f8106f64640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f8106f64640 10 monclient: _renew_subs 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f8106f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81001344f0 con 0x7f81000b32b0 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.378+0000 7f8106f64640 1 librados: init done 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.380+0000 7f8106f64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.380+0000 7f8106f64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.380+0000 7f8106f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.380+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81001398b0 con 0x7f81000b32b0 2024-03-29T10:51:20.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.380+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80fc001a50 con 0x7f81000b32b0 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.380+0000 7f80f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.380+0000 7f80f7fff640 4 set_mon_vals no callback set 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.382+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f80fc004a40 con 0x7f81000b32b0 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.382+0000 7f80f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.382+0000 7f80f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.382+0000 7f80f67fc640 10 monclient: dump: 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:20.378 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.382+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f80fc01aad0 con 0x7f81000b32b0 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.383+0000 7f8105761640 10 monclient: get_auth_request con 0x7f80e406e140 auth_method 0 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.383+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f80fc0849c0 con 0x7f81000b32b0 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.383+0000 7f80f67fc640 10 client.4385.objecter ms_dispatch 0x7f8100000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.383+0000 7f80f67fc640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.383+0000 7f80f67fc640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.384+0000 7f80f67fc640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T10:51:20.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.384+0000 7f80f67fc640 10 client.4385.objecter ms_handle_connect 0x7f80e406e140 2024-03-29T10:51:20.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.391+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80fc050190 con 0x7f81000b32b0 2024-03-29T10:51:20.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.391+0000 7f80f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:20.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.391+0000 7f80f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:20.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.759+0000 7f8106f64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:20.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.759+0000 7f8106f64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:20.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.759+0000 7f8106f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:20.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.759+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f810004f730 con 0x7f81000b32b0 2024-03-29T10:51:20.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.760+0000 7f80f67fc640 1 -- 172.21.15.5:0/4190069983 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f80fc052290 con 0x7f81000b32b0 2024-03-29T10:51:20.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.760+0000 7f80f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:20.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.760+0000 7f80f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:20.742 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:51:20.742 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:20.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 10 librados: watch_flush enter 2024-03-29T10:51:20.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 10 librados: watch_flush exit 2024-03-29T10:51:20.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T10:51:20.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 10 client.4385.objecter successfully canceled tick 2024-03-29T10:51:20.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 >> v1:172.21.15.5:6816/2847500151 conn(0x7f80e406e140 legacy=0x7f80e4070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 10 monclient: shutdown 2024-03-29T10:51:20.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:20.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 >> v1:172.21.15.5:6790/0 conn(0x7f81000b32b0 legacy=0x7f8100133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:20.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 shutdown_connections 2024-03-29T10:51:20.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 >> 172.21.15.5:0/4190069983 conn(0x7f810006d140 msgr2=0x7f81000b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:20.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 shutdown_connections 2024-03-29T10:51:20.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.764+0000 7f8106f64640 1 -- 172.21.15.5:0/4190069983 wait complete. 2024-03-29T10:51:20.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:20.765+0000 7f8106f64640 1 librados: shutdown 2024-03-29T10:51:20.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:27.568 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:27.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.721+0000 7f1726388640 10 monclient: get_monmap_and_config 2024-03-29T10:51:27.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.721+0000 7f1726388640 10 monclient: build_initial_monmap 2024-03-29T10:51:27.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.722+0000 7f1726388640 10 monclient: monmap: 2024-03-29T10:51:27.704 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:27.704 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:27.704 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:51:27.723024+0000 2024-03-29T10:51:27.704 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:51:27.723024+0000 2024-03-29T10:51:27.704 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:27.704 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:27.704 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:27.704 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:51:27.704 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:51:27.704 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:27.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.722+0000 7f1726388640 1 Processor -- start 2024-03-29T10:51:27.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.723+0000 7f1726388640 1 -- start start 2024-03-29T10:51:27.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.723+0000 7f1726388640 10 monclient: init 2024-03-29T10:51:27.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.723+0000 7f1726388640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:27.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.723+0000 7f1726388640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:27.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f1726388640 10 monclient(hunting): picked mon.noname-b con 0x7f1720070170 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:27.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f171ffff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f1720070170 0x7f172013d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:57224/0 (socket says 172.21.15.5:57224) 2024-03-29T10:51:27.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f171ffff640 1 -- 172.21.15.5:0/2098767511 learned_addr learned my addr 172.21.15.5:0/2098767511 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:27.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f1726388640 10 monclient(hunting): picked mon.noname-a con 0x7f17200713f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:27.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f1726388640 10 monclient(hunting): picked mon.noname-c con 0x7f1720070550 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:27.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f1726388640 1 -- 172.21.15.5:0/2098767511 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172013dbd0 con 0x7f17200713f0 2024-03-29T10:51:27.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f1726388640 1 -- 172.21.15.5:0/2098767511 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172013ff70 con 0x7f1720070550 2024-03-29T10:51:27.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f1726388640 1 -- 172.21.15.5:0/2098767511 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1720140250 con 0x7f1720070170 2024-03-29T10:51:27.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f1726388640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:27.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f1726388640 10 monclient(hunting): authenticate will time out at 1100.965515s 2024-03-29T10:51:27.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f171effd640 1 -- 172.21.15.5:0/2098767511 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841749630 0 0) 0x7f1720140250 con 0x7f1720070170 2024-03-29T10:51:27.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f171effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:27.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f171effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:27.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f171effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:27.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f171effd640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:51:27.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.724+0000 7f171effd640 1 -- 172.21.15.5:0/2098767511 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1708003520 con 0x7f1720070170 2024-03-29T10:51:27.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.725+0000 7f171effd640 1 -- 172.21.15.5:0/2098767511 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552005425 0 0) 0x7f1708003520 con 0x7f1720070170 2024-03-29T10:51:27.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.725+0000 7f171effd640 1 -- 172.21.15.5:0/2098767511 >> v1:172.21.15.177:6789/0 conn(0x7f1720070550 legacy=0x7f172014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:27.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.725+0000 7f171effd640 1 -- 172.21.15.5:0/2098767511 >> v1:172.21.15.5:6789/0 conn(0x7f17200713f0 legacy=0x7f17201406e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 1 monclient: found mon.noname-b 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 1 -- 172.21.15.5:0/2098767511 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1720140df0 con 0x7f1720070170 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: _finish_auth 0 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 1 -- 172.21.15.5:0/2098767511 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f1710001690 con 0x7f1720070170 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: dump: 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:27.709 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f1726388640 5 monclient: authenticate success, global_id 4388 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f1726388640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 1 -- 172.21.15.5:0/2098767511 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1710001920 con 0x7f1720070170 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:27.710 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 1 -- 172.21.15.5:0/2098767511 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f1710001cb0 con 0x7f1720070170 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f171effd640 10 monclient: dump: 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f1726388640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:27.711 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.727+0000 7f1726388640 4 set_mon_vals no callback set 2024-03-29T10:51:27.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.728+0000 7f1726388640 10 monclient: shutdown 2024-03-29T10:51:27.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.728+0000 7f1726388640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:27.713 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.728+0000 7f1726388640 1 -- 172.21.15.5:0/2098767511 >> v1:172.21.15.5:6790/0 conn(0x7f1720070170 legacy=0x7f172013d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.728+0000 7f1726388640 1 -- 172.21.15.5:0/2098767511 shutdown_connections 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.728+0000 7f1726388640 1 -- 172.21.15.5:0/2098767511 >> 172.21.15.5:0/2098767511 conn(0x7f1720067610 msgr2=0x7f17200b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.728+0000 7f1726388640 1 -- 172.21.15.5:0/2098767511 shutdown_connections 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.728+0000 7f1726388640 1 -- 172.21.15.5:0/2098767511 wait complete. 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.729+0000 7f1726388640 10 monclient: build_initial_monmap 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.729+0000 7f1726388640 10 monclient: monmap: 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:27.714 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.729+0000 7f1726388640 1 librados: starting msgr at 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.729+0000 7f1726388640 1 librados: starting objecter 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.729+0000 7f1726388640 1 Processor -- start 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.730+0000 7f1726388640 1 -- start start 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.730+0000 7f1726388640 1 librados: setting wanted keys 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.730+0000 7f1726388640 1 librados: calling monclient init 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.730+0000 7f1726388640 10 monclient: init 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.730+0000 7f1726388640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.730+0000 7f1726388640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.730+0000 7f1726388640 10 monclient(hunting): picked mon.noname-c con 0x7f1720070170 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.730+0000 7f1726388640 10 monclient(hunting): picked mon.noname-a con 0x7f1720070550 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.731+0000 7f1726388640 10 monclient(hunting): picked mon.noname-b con 0x7f17200713f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:27.715 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.731+0000 7f171ffff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f1720070170 0x7f1720138210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:57440/0 (socket says 172.21.15.5:57440) 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.731+0000 7f171ffff640 1 -- 172.21.15.5:0/1826246511 learned_addr learned my addr 172.21.15.5:0/1826246511 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.731+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17201d7290 con 0x7f1720070550 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.731+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1720138920 con 0x7f17200713f0 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.731+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17201d7e30 con 0x7f1720070170 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.731+0000 7f171cff9640 10 client.?.objecter ms_handle_connect 0x7f1720070170 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.731+0000 7f171cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.731+0000 7f1726388640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.731+0000 7f1726388640 10 monclient(hunting): authenticate will time out at 1100.972655s 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276027646 0 0) 0x7f17201d7e30 con 0x7f1720070170 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:51:27.716 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f170c004eb0 con 0x7f1720070170 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3612346901 0 0) 0x7f170c004eb0 con 0x7f1720070170 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17201d7e30 con 0x7f1720070170 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f1710001610 con 0x7f1720070170 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:27.718 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 10 monclient(hunting): dump: 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.732+0000 7f171cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.733+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3448638917 0 0) 0x7f17201d7e30 con 0x7f1720070170 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.733+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 >> v1:172.21.15.177:6789/0 conn(0x7f17200713f0 legacy=0x7f17201e1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:27.719 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.733+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 >> v1:172.21.15.5:6789/0 conn(0x7f1720070550 legacy=0x7f17201d7720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 1 monclient: found mon.c 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17200b3610 con 0x7f1720070170 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f1726388640 5 monclient: authenticate success, global_id 4391 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f1726388640 10 monclient: _renew_subs 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f1726388640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17200b3860 con 0x7f1720070170 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1710004a90 con 0x7f1720070170 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:27.720 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f1710004de0 con 0x7f1720070170 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171cff9640 10 monclient: dump: 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f171e7fc640 4 set_mon_vals no callback set 2024-03-29T10:51:27.721 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f1726388640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.734+0000 7f1726388640 10 monclient: _renew_subs 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.735+0000 7f1726388640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.735+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1720138de0 con 0x7f1720070170 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.735+0000 7f1726388640 1 librados: init done 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.736+0000 7f1726388640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.736+0000 7f1726388640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.736+0000 7f1726388640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.736+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17201d7060 con 0x7f1720070170 2024-03-29T10:51:27.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.737+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f171001adb0 con 0x7f1720070170 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.737+0000 7f171f7fe640 10 monclient: get_auth_request con 0x7f170c06dca0 auth_method 0 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.738+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f1710084b70 con 0x7f1720070170 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.738+0000 7f171cff9640 10 client.4391.objecter ms_dispatch 0x7f1720000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.738+0000 7f171cff9640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.738+0000 7f171cff9640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.738+0000 7f171cff9640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-29T10:51:27.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.738+0000 7f171cff9640 10 client.4391.objecter ms_handle_connect 0x7f170c06dca0 2024-03-29T10:51:27.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.747+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17201d7060 con 0x7f1720070170 2024-03-29T10:51:27.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.747+0000 7f171cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:27.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:27.747+0000 7f171cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:28.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.059+0000 7f1726388640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:28.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.059+0000 7f1726388640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:28.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.059+0000 7f1726388640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:28.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.059+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17201d7060 con 0x7f1720070170 2024-03-29T10:51:28.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.062+0000 7f171cff9640 1 -- 172.21.15.5:0/1826246511 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f17201d7060 con 0x7f1720070170 2024-03-29T10:51:28.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.062+0000 7f171cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:28.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.062+0000 7f171cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:28.044 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:51:28.044 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:28.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.064+0000 7f1726388640 10 librados: watch_flush enter 2024-03-29T10:51:28.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 10 librados: watch_flush exit 2024-03-29T10:51:28.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-29T10:51:28.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 10 client.4391.objecter successfully canceled tick 2024-03-29T10:51:28.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 >> v1:172.21.15.5:6816/2847500151 conn(0x7f170c06dca0 legacy=0x7f170c070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:28.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 10 monclient: shutdown 2024-03-29T10:51:28.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:28.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 >> v1:172.21.15.5:6790/0 conn(0x7f1720070170 legacy=0x7f1720138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:28.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 shutdown_connections 2024-03-29T10:51:28.048 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 >> 172.21.15.5:0/1826246511 conn(0x7f1720067610 msgr2=0x7f172013d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:28.048 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 shutdown_connections 2024-03-29T10:51:28.048 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.065+0000 7f1726388640 1 -- 172.21.15.5:0/1826246511 wait complete. 2024-03-29T10:51:28.048 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:28.066+0000 7f1726388640 1 librados: shutdown 2024-03-29T10:51:28.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:34.869 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:35.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.036+0000 7fe9ae443640 10 monclient: get_monmap_and_config 2024-03-29T10:51:35.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.036+0000 7fe9ae443640 10 monclient: build_initial_monmap 2024-03-29T10:51:35.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.036+0000 7fe9ae443640 10 monclient: monmap: 2024-03-29T10:51:35.018 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:35.018 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:35.018 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:51:35.037675+0000 2024-03-29T10:51:35.019 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:51:35.037675+0000 2024-03-29T10:51:35.019 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:35.019 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:35.019 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:35.019 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:51:35.019 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:51:35.019 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:35.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.037+0000 7fe9ae443640 1 Processor -- start 2024-03-29T10:51:35.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.037+0000 7fe9ae443640 1 -- start start 2024-03-29T10:51:35.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.037+0000 7fe9ae443640 10 monclient: init 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.038+0000 7fe9ae443640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.038+0000 7fe9ae443640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.038+0000 7fe9ae443640 10 monclient(hunting): picked mon.noname-a con 0x7fe9a0006890 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.038+0000 7fe9ae443640 10 monclient(hunting): picked mon.noname-c con 0x7fe9a0006c70 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.038+0000 7fe9ae443640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a002b4c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.038+0000 7fe9ae443640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a002c830 con 0x7fe9a0006890 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.038+0000 7fe9ae443640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a002afa0 con 0x7fe9a0006c70 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.038+0000 7fe9ae443640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a002aba0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.038+0000 7fe9ae443640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.038+0000 7fe9ae443640 10 monclient(hunting): authenticate will time out at 1108.279753s 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.041+0000 7fe9adc42640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fe9a002b4c0 0x7fe9a00cf0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:58512/0 (socket says 172.21.15.5:58512) 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.041+0000 7fe9adc42640 1 -- 172.21.15.5:0/1106971516 learned_addr learned my addr 172.21.15.5:0/1106971516 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:35.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.041+0000 7fe9ad441640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fe9a0006890 0x7fe9a00c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:55157/0 (socket says 172.21.15.5:55157) 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.041+0000 7fe99ffff640 1 -- 172.21.15.5:0/1106971516 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259640596 0 0) 0x7fe9a002aba0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.041+0000 7fe99ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.041+0000 7fe99ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.041+0000 7fe99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.041+0000 7fe99ffff640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:51:35.024 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.041+0000 7fe99ffff640 1 -- 172.21.15.5:0/1106971516 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe994003520 con 0x7fe9a002b4c0 2024-03-29T10:51:35.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.043+0000 7fe99ffff640 1 -- 172.21.15.5:0/1106971516 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3692234749 0 0) 0x7fe994003520 con 0x7fe9a002b4c0 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.043+0000 7fe99ffff640 1 -- 172.21.15.5:0/1106971516 >> v1:172.21.15.177:6789/0 conn(0x7fe9a0006c70 legacy=0x7fe9a00c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.043+0000 7fe99ffff640 1 -- 172.21.15.5:0/1106971516 >> v1:172.21.15.5:6789/0 conn(0x7fe9a0006890 legacy=0x7fe9a00c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.043+0000 7fe99ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.043+0000 7fe99ffff640 1 monclient: found mon.noname-b 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.043+0000 7fe99ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.043+0000 7fe99ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.043+0000 7fe99ffff640 1 -- 172.21.15.5:0/1106971516 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a002ad30 con 0x7fe9a002b4c0 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.043+0000 7fe99ffff640 10 monclient: _finish_auth 0 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 1 -- 172.21.15.5:0/1106971516 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe9a804f730 con 0x7fe9a002b4c0 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:35.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 10 monclient: dump: 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 1 -- 172.21.15.5:0/1106971516 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9a80008d0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 1 -- 172.21.15.5:0/1106971516 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe9a80758b0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.044+0000 7fe99ffff640 10 monclient: dump: 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:35.029 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:35.030 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:35.030 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:35.030 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.045+0000 7fe9ae443640 5 monclient: authenticate success, global_id 4394 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.045+0000 7fe9ae443640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.045+0000 7fe9ae443640 4 set_mon_vals no callback set 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.046+0000 7fe9ae443640 10 monclient: shutdown 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.046+0000 7fe9ae443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:35.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.046+0000 7fe9ae443640 1 -- 172.21.15.5:0/1106971516 >> v1:172.21.15.5:6790/0 conn(0x7fe9a002b4c0 legacy=0x7fe9a00cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.046+0000 7fe9ae443640 1 -- 172.21.15.5:0/1106971516 shutdown_connections 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.046+0000 7fe9ae443640 1 -- 172.21.15.5:0/1106971516 >> 172.21.15.5:0/1106971516 conn(0x7fe9a00058f0 msgr2=0x7fe9a0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.046+0000 7fe9ae443640 1 -- 172.21.15.5:0/1106971516 shutdown_connections 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.047+0000 7fe9ae443640 1 -- 172.21.15.5:0/1106971516 wait complete. 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.048+0000 7fe9ae443640 10 monclient: build_initial_monmap 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.048+0000 7fe9ae443640 10 monclient: monmap: 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:51:35.032 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:35.033 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:35.033 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:35.033 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:51:35.033 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:51:35.033 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:35.033 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.048+0000 7fe9ae443640 1 librados: starting msgr at 2024-03-29T10:51:35.033 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.048+0000 7fe9ae443640 1 librados: starting objecter 2024-03-29T10:51:35.033 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.049+0000 7fe9ae443640 1 Processor -- start 2024-03-29T10:51:35.033 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.049+0000 7fe9ae443640 1 -- start start 2024-03-29T10:51:35.033 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.049+0000 7fe9ae443640 1 librados: setting wanted keys 2024-03-29T10:51:35.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.049+0000 7fe9ae443640 1 librados: calling monclient init 2024-03-29T10:51:35.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.049+0000 7fe9ae443640 10 monclient: init 2024-03-29T10:51:35.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.049+0000 7fe9ae443640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:35.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.050+0000 7fe9ae443640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe9ae443640 10 monclient(hunting): picked mon.noname-a con 0x7fe9a0006890 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe9ae443640 10 monclient(hunting): picked mon.noname-c con 0x7fe9a002b4c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe9ae443640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a00c6f70 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe9acc40640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fe9a002b4c0 0x7fe9a0160230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:60236/0 (socket says 172.21.15.5:60236) 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe9acc40640 1 -- 172.21.15.5:0/3565127216 learned_addr learned my addr 172.21.15.5:0/3565127216 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00be000 con 0x7fe9a0006890 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00c6220 con 0x7fe9a00c6f70 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00bfaf0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe9ae443640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe9ae443640 10 monclient(hunting): authenticate will time out at 1108.292948s 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe99dffb640 10 client.?.objecter ms_handle_connect 0x7fe9a002b4c0 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.051+0000 7fe99dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:35.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103222123 0 0) 0x7fe9a00bfaf0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98c004eb0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2852114482 0 0) 0x7fe98c004eb0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a00bfaf0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe9a4001970 con 0x7fe9a002b4c0 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 10 monclient(hunting): dump: 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:35.038 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1164154758 0 0) 0x7fe9a00bfaf0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 >> v1:172.21.15.177:6789/0 conn(0x7fe9a00c6f70 legacy=0x7fe9a0160940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.052+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 >> v1:172.21.15.5:6789/0 conn(0x7fe9a0006890 legacy=0x7fe9a00bc250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe99dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe99dffb640 1 monclient: found mon.c 2024-03-29T10:51:35.039 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe99dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe99dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a00bf760 con 0x7fe9a002b4c0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe99dffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe99dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe99dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe9ae443640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe9ae443640 10 monclient: _renew_subs 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe9ae443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.053+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a00bc960 con 0x7fe9a002b4c0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.054+0000 7fe9ae443640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.054+0000 7fe9ae443640 10 monclient: _renew_subs 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.054+0000 7fe9ae443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.054+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a00bcf60 con 0x7fe9a002b4c0 2024-03-29T10:51:35.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.054+0000 7fe9ae443640 1 librados: init done 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.057+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9a4001c70 con 0x7fe9a002b4c0 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.057+0000 7fe99dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.057+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe9a4003a60 con 0x7fe9a002b4c0 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.057+0000 7fe99dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.057+0000 7fe99dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.057+0000 7fe99dffb640 10 monclient: dump: 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:35.041 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.057+0000 7fe99effd640 4 set_mon_vals no callback set 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.058+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fe9a4003c80 con 0x7fe9a002b4c0 2024-03-29T10:51:35.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.059+0000 7fe9ae443640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.059+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fe9a4084b10 con 0x7fe9a002b4c0 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.059+0000 7fe99dffb640 10 client.4397.objecter ms_dispatch 0x7fe9a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.059+0000 7fe99dffb640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.059+0000 7fe99dffb640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.059+0000 7fe99dffb640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.060+0000 7fe9ae443640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.060+0000 7fe9ae443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.060+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe988005080 con 0x7fe9a002b4c0 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.060+0000 7fe9ad441640 10 monclient: get_auth_request con 0x7fe98c06dc90 auth_method 0 2024-03-29T10:51:35.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.061+0000 7fe99dffb640 10 client.4397.objecter ms_handle_connect 0x7fe98c06dc90 2024-03-29T10:51:35.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.071+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9a40503f0 con 0x7fe9a002b4c0 2024-03-29T10:51:35.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.071+0000 7fe99dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:35.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.071+0000 7fe99dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:35.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.398+0000 7fe9ae443640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:35.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.398+0000 7fe9ae443640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:35.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.398+0000 7fe9ae443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:35.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.398+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe988005330 con 0x7fe9a002b4c0 2024-03-29T10:51:35.382 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.401+0000 7fe99dffb640 1 -- 172.21.15.5:0/3565127216 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fe9a4052320 con 0x7fe9a002b4c0 2024-03-29T10:51:35.382 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.401+0000 7fe99dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:35.382 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.401+0000 7fe99dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:35.383 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:51:35.383 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:35.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.404+0000 7fe9ae443640 10 librados: watch_flush enter 2024-03-29T10:51:35.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.406+0000 7fe9ae443640 10 librados: watch_flush exit 2024-03-29T10:51:35.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.406+0000 7fe9ae443640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T10:51:35.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.407+0000 7fe9ae443640 10 client.4397.objecter successfully canceled tick 2024-03-29T10:51:35.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.407+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 >> v1:172.21.15.5:6816/2847500151 conn(0x7fe98c06dc90 legacy=0x7fe98c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.407+0000 7fe9ae443640 10 monclient: shutdown 2024-03-29T10:51:35.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.407+0000 7fe9ae443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:35.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.407+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 >> v1:172.21.15.5:6790/0 conn(0x7fe9a002b4c0 legacy=0x7fe9a0160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:35.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.407+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 shutdown_connections 2024-03-29T10:51:35.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.407+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 >> 172.21.15.5:0/3565127216 conn(0x7fe9a00058f0 msgr2=0x7fe9a0038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:35.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.409+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 shutdown_connections 2024-03-29T10:51:35.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.409+0000 7fe9ae443640 1 -- 172.21.15.5:0/3565127216 wait complete. 2024-03-29T10:51:35.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:35.409+0000 7fe9ae443640 1 librados: shutdown 2024-03-29T10:51:35.411 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:42.219 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.389+0000 7f9089ec5640 10 monclient: get_monmap_and_config 2024-03-29T10:51:42.373 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.389+0000 7f9089ec5640 10 monclient: build_initial_monmap 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.389+0000 7f9089ec5640 10 monclient: monmap: 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:51:42.390276+0000 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:51:42.390276+0000 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:51:42.374 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:42.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.392+0000 7f9089ec5640 1 Processor -- start 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.392+0000 7f9089ec5640 1 -- start start 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.392+0000 7f9089ec5640 10 monclient: init 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.393+0000 7f9089ec5640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.393+0000 7f9089ec5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.393+0000 7f9089ec5640 10 monclient(hunting): picked mon.noname-a con 0x7f90840a9ac0 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.393+0000 7f9089ec5640 10 monclient(hunting): picked mon.noname-b con 0x7f908413ee90 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.393+0000 7f9089ec5640 10 monclient(hunting): picked mon.noname-c con 0x7f90840af900 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.393+0000 7f9089ec5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840afd10 con 0x7f90840a9ac0 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.393+0000 7f9089ec5640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840af6f0 con 0x7f90840af900 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.393+0000 7f9089ec5640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840aac80 con 0x7f908413ee90 2024-03-29T10:51:42.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.393+0000 7f9089ec5640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:42.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.393+0000 7f9089ec5640 10 monclient(hunting): authenticate will time out at 1115.634977s 2024-03-29T10:51:42.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.396+0000 7f90837fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f90840a9ac0 0x7f90840adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45537/0 (socket says 172.21.15.5:45537) 2024-03-29T10:51:42.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.396+0000 7f90837fe640 1 -- 172.21.15.5:0/1810902458 learned_addr learned my addr 172.21.15.5:0/1810902458 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:42.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.396+0000 7f90827fc640 1 -- 172.21.15.5:0/1810902458 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159237528 0 0) 0x7f90840afd10 con 0x7f90840a9ac0 2024-03-29T10:51:42.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.396+0000 7f90827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:42.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.396+0000 7f90827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:42.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.396+0000 7f90827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:42.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.396+0000 7f90827fc640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:51:42.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.396+0000 7f90827fc640 1 -- 172.21.15.5:0/1810902458 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f906c003500 con 0x7f90840a9ac0 2024-03-29T10:51:42.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.397+0000 7f90827fc640 1 -- 172.21.15.5:0/1810902458 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339893274 0 0) 0x7f906c003500 con 0x7f90840a9ac0 2024-03-29T10:51:42.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.397+0000 7f90827fc640 1 -- 172.21.15.5:0/1810902458 >> v1:172.21.15.5:6790/0 conn(0x7f908413ee90 legacy=0x7f908413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 1 -- 172.21.15.5:0/1810902458 >> v1:172.21.15.177:6789/0 conn(0x7f90840af900 legacy=0x7f9084149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 1 monclient: found mon.noname-a 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 1 -- 172.21.15.5:0/1810902458 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90840aaf60 con 0x7f90840a9ac0 2024-03-29T10:51:42.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 1 -- 172.21.15.5:0/1810902458 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9074004cb0 con 0x7f90840a9ac0 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: dump: 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:42.385 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 1 -- 172.21.15.5:0/1810902458 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9074004e80 con 0x7f90840a9ac0 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 1 -- 172.21.15.5:0/1810902458 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f90740013f0 con 0x7f90840a9ac0 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:42.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.398+0000 7f90827fc640 10 monclient: dump: 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.401+0000 7f9089ec5640 5 monclient: authenticate success, global_id 4347 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.401+0000 7f9089ec5640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.401+0000 7f9089ec5640 4 set_mon_vals no callback set 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.401+0000 7f9089ec5640 10 monclient: shutdown 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.401+0000 7f9089ec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:42.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.401+0000 7f9089ec5640 1 -- 172.21.15.5:0/1810902458 >> v1:172.21.15.5:6789/0 conn(0x7f90840a9ac0 legacy=0x7f90840adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.402+0000 7f9089ec5640 1 -- 172.21.15.5:0/1810902458 shutdown_connections 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.402+0000 7f9089ec5640 1 -- 172.21.15.5:0/1810902458 >> 172.21.15.5:0/1810902458 conn(0x7f90840b35e0 msgr2=0x7f90840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.402+0000 7f9089ec5640 1 -- 172.21.15.5:0/1810902458 shutdown_connections 2024-03-29T10:51:42.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.402+0000 7f9089ec5640 1 -- 172.21.15.5:0/1810902458 wait complete. 2024-03-29T10:51:42.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.403+0000 7f9089ec5640 10 monclient: build_initial_monmap 2024-03-29T10:51:42.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.403+0000 7f9089ec5640 10 monclient: monmap: 2024-03-29T10:51:42.389 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:42.389 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.403+0000 7f9089ec5640 1 librados: starting msgr at 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.403+0000 7f9089ec5640 1 librados: starting objecter 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.403+0000 7f9089ec5640 1 Processor -- start 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.404+0000 7f9089ec5640 1 -- start start 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.404+0000 7f9089ec5640 1 librados: setting wanted keys 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.404+0000 7f9089ec5640 1 librados: calling monclient init 2024-03-29T10:51:42.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.404+0000 7f9089ec5640 10 monclient: init 2024-03-29T10:51:42.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.405+0000 7f9089ec5640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:42.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.405+0000 7f9089ec5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:42.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.405+0000 7f9089ec5640 10 monclient(hunting): picked mon.noname-a con 0x7f90840a9ac0 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:42.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.405+0000 7f9089ec5640 10 monclient(hunting): picked mon.noname-c con 0x7f90840af900 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.411+0000 7f9082ffd640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f90840af900 0x7f908413f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:43003/0 (socket says 172.21.15.5:43003) 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f9083fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f908413ee90 0x7f90841e6100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:36483/0 (socket says 172.21.15.5:36483) 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f9089ec5640 10 monclient(hunting): picked mon.noname-b con 0x7f908413ee90 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f9089ec5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9084142620 con 0x7f90840a9ac0 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f9089ec5640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90840700a0 con 0x7f908413ee90 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f9089ec5640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9084070380 con 0x7f90840af900 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f9089ec5640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f9089ec5640 10 monclient(hunting): authenticate will time out at 1115.653630s 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f9082ffd640 1 -- 172.21.15.5:0/1675162328 learned_addr learned my addr 172.21.15.5:0/1675162328 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f906bfff640 10 client.?.objecter ms_handle_connect 0x7f90840af900 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f906bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800271996 0 0) 0x7f9084070380 con 0x7f90840af900 2024-03-29T10:51:42.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f906bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f906bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f906bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f906bfff640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.412+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9070004eb0 con 0x7f90840af900 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 10 client.?.objecter ms_handle_connect 0x7f90840a9ac0 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4076283296 0 0) 0x7f9070004eb0 con 0x7f90840af900 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9084070380 con 0x7f90840af900 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f907c001c10 con 0x7f90840af900 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 10 monclient(hunting): dump: 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:42.401 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903520055 0 0) 0x7f9084142620 con 0x7f90840a9ac0 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:51:42.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9070004eb0 con 0x7f90840a9ac0 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 473300888 0 0) 0x7f9084070380 con 0x7f90840af900 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.413+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 >> v1:172.21.15.177:6789/0 conn(0x7f908413ee90 legacy=0x7f90841e6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 >> v1:172.21.15.5:6789/0 conn(0x7f90840a9ac0 legacy=0x7f9084070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 1 monclient: found mon.c 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9084070660 con 0x7f90840af900 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 10 monclient: _finish_auth 0 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:42.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 10 client.?.objecter ms_handle_connect 0x7f908413ee90 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f906bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f9089ec5640 5 monclient: authenticate success, global_id 4403 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f9089ec5640 10 monclient: _renew_subs 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f9089ec5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f9089ec5640 1 -- 172.21.15.5:0/1675162328 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9084071500 con 0x7f90840af900 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f9089ec5640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f9089ec5640 10 monclient: _renew_subs 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f9089ec5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f9089ec5640 1 -- 172.21.15.5:0/1675162328 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f908406e980 con 0x7f90840af900 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.414+0000 7f9089ec5640 1 librados: init done 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.415+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f907c003990 con 0x7f90840af900 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.415+0000 7f906bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.415+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f907c003d20 con 0x7f90840af900 2024-03-29T10:51:42.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.415+0000 7f906bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.415+0000 7f906bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.415+0000 7f906bfff640 10 monclient: dump: 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.415+0000 7f9081ffb640 4 set_mon_vals no callback set 2024-03-29T10:51:42.405 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.416+0000 7f9089ec5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:42.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.416+0000 7f9089ec5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:42.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.416+0000 7f9089ec5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:42.406 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.416+0000 7f9089ec5640 1 -- 172.21.15.5:0/1675162328 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9064005080 con 0x7f90840af900 2024-03-29T10:51:42.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.420+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f907c003e80 con 0x7f90840af900 2024-03-29T10:51:42.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.421+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f907c084cc0 con 0x7f90840af900 2024-03-29T10:51:42.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.421+0000 7f906bfff640 10 client.4403.objecter ms_dispatch 0x7f9084000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:42.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.421+0000 7f906bfff640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:42.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.421+0000 7f906bfff640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:42.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.421+0000 7f906bfff640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T10:51:42.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.421+0000 7f90837fe640 10 monclient: get_auth_request con 0x7f907006db00 auth_method 0 2024-03-29T10:51:42.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.422+0000 7f906bfff640 10 client.4403.objecter ms_handle_connect 0x7f907006db00 2024-03-29T10:51:42.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.427+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f907c0505a0 con 0x7f90840af900 2024-03-29T10:51:42.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.427+0000 7f906bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:42.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.427+0000 7f906bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:42.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.758+0000 7f9089ec5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:42.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.758+0000 7f9089ec5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:42.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.758+0000 7f9089ec5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:42.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.758+0000 7f9089ec5640 1 -- 172.21.15.5:0/1675162328 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9064005330 con 0x7f90840af900 2024-03-29T10:51:42.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.760+0000 7f906bfff640 1 -- 172.21.15.5:0/1675162328 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f907c0524d0 con 0x7f90840af900 2024-03-29T10:51:42.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.760+0000 7f906bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:42.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.760+0000 7f906bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:42.741 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:51:42.742 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:42.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.763+0000 7f9089ec5640 10 librados: watch_flush enter 2024-03-29T10:51:42.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.764+0000 7f9089ec5640 10 librados: watch_flush exit 2024-03-29T10:51:42.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.764+0000 7f9089ec5640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T10:51:42.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.764+0000 7f9089ec5640 10 client.4403.objecter successfully canceled tick 2024-03-29T10:51:42.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.765+0000 7f9089ec5640 1 -- 172.21.15.5:0/1675162328 >> v1:172.21.15.5:6816/2847500151 conn(0x7f907006db00 legacy=0x7f907006ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:42.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.765+0000 7f9089ec5640 10 monclient: shutdown 2024-03-29T10:51:42.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.765+0000 7f9089ec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:42.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.765+0000 7f9089ec5640 1 -- 172.21.15.5:0/1675162328 >> v1:172.21.15.5:6790/0 conn(0x7f90840af900 legacy=0x7f908413f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:42.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.768+0000 7f9089ec5640 1 -- 172.21.15.5:0/1675162328 shutdown_connections 2024-03-29T10:51:42.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.768+0000 7f9089ec5640 1 -- 172.21.15.5:0/1675162328 >> 172.21.15.5:0/1675162328 conn(0x7f90840b35e0 msgr2=0x7f908413b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:42.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.769+0000 7f9089ec5640 1 -- 172.21.15.5:0/1675162328 shutdown_connections 2024-03-29T10:51:42.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.769+0000 7f9089ec5640 1 -- 172.21.15.5:0/1675162328 wait complete. 2024-03-29T10:51:42.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:42.769+0000 7f9089ec5640 1 librados: shutdown 2024-03-29T10:51:42.769 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:49.576 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:49.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.741+0000 7fa5b19c4640 10 monclient: get_monmap_and_config 2024-03-29T10:51:49.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.741+0000 7fa5b19c4640 10 monclient: build_initial_monmap 2024-03-29T10:51:49.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.741+0000 7fa5b19c4640 10 monclient: monmap: 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:51:49.742722+0000 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:51:49.742722+0000 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.742+0000 7fa5b19c4640 1 Processor -- start 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.742+0000 7fa5b19c4640 1 -- start start 2024-03-29T10:51:49.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.742+0000 7fa5b19c4640 10 monclient: init 2024-03-29T10:51:49.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.743+0000 7fa5b19c4640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:49.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.743+0000 7fa5b19c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:49.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.743+0000 7fa5b19c4640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ac1457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:49.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.743+0000 7fa5b19c4640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ac13f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.744+0000 7fa5ab7fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fa5ac0a9d70 0x7fa5ac14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:38800/0 (socket says 172.21.15.5:38800) 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.744+0000 7fa5ab7fe640 1 -- 172.21.15.5:0/3312550538 learned_addr learned my addr 172.21.15.5:0/3312550538 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.744+0000 7fa5b19c4640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ac0a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.744+0000 7fa5b19c4640 1 -- 172.21.15.5:0/3312550538 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0aaef0 con 0x7fa5ac13f650 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.744+0000 7fa5b19c4640 1 -- 172.21.15.5:0/3312550538 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac145bd0 con 0x7fa5ac1457f0 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.744+0000 7fa5b19c4640 1 -- 172.21.15.5:0/3312550538 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac145eb0 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.744+0000 7fa5b19c4640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:49.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.744+0000 7fa5b19c4640 10 monclient(hunting): authenticate will time out at 1122.985979s 2024-03-29T10:51:49.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.746+0000 7fa5a9ffb640 1 -- 172.21.15.5:0/3312550538 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209394194 0 0) 0x7fa5ac145eb0 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.746+0000 7fa5a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:49.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.746+0000 7fa5a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:49.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.746+0000 7fa5a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:49.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.746+0000 7fa5a9ffb640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:51:49.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.746+0000 7fa5a9ffb640 1 -- 172.21.15.5:0/3312550538 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa594003520 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.747+0000 7fa5a9ffb640 1 -- 172.21.15.5:0/3312550538 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 720875369 0 0) 0x7fa594003520 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.728 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.747+0000 7fa5a9ffb640 1 -- 172.21.15.5:0/3312550538 >> v1:172.21.15.177:6789/0 conn(0x7fa5ac1457f0 legacy=0x7fa5ac13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 1 -- 172.21.15.5:0/3312550538 >> v1:172.21.15.5:6789/0 conn(0x7fa5ac13f650 legacy=0x7fa5ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 1 monclient: found mon.noname-b 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 1 -- 172.21.15.5:0/3312550538 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac146340 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 1 -- 172.21.15.5:0/3312550538 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa5a4001a10 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5a9ffb640 10 monclient: dump: 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5b19c4640 5 monclient: authenticate success, global_id 4406 2024-03-29T10:51:49.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.750+0000 7fa5b19c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:49.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.751+0000 7fa5a9ffb640 1 -- 172.21.15.5:0/3312550538 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5a4001be0 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.751+0000 7fa5a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:49.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.751+0000 7fa5b19c4640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:49.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.751+0000 7fa5b19c4640 4 set_mon_vals no callback set 2024-03-29T10:51:49.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.751+0000 7fa5a9ffb640 1 -- 172.21.15.5:0/3312550538 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa5a4001ff0 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.751+0000 7fa5a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:49.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.751+0000 7fa5a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.751+0000 7fa5a9ffb640 10 monclient: dump: 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.752+0000 7fa5b19c4640 10 monclient: shutdown 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.752+0000 7fa5b19c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.752+0000 7fa5b19c4640 1 -- 172.21.15.5:0/3312550538 >> v1:172.21.15.5:6790/0 conn(0x7fa5ac0a9d70 legacy=0x7fa5ac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.752+0000 7fa5b19c4640 1 -- 172.21.15.5:0/3312550538 shutdown_connections 2024-03-29T10:51:49.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.752+0000 7fa5b19c4640 1 -- 172.21.15.5:0/3312550538 >> 172.21.15.5:0/3312550538 conn(0x7fa5ac066a00 msgr2=0x7fa5ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.752+0000 7fa5b19c4640 1 -- 172.21.15.5:0/3312550538 shutdown_connections 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.752+0000 7fa5b19c4640 1 -- 172.21.15.5:0/3312550538 wait complete. 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.753+0000 7fa5b19c4640 10 monclient: build_initial_monmap 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.753+0000 7fa5b19c4640 10 monclient: monmap: 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:49.736 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.754+0000 7fa5b19c4640 1 librados: starting msgr at 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.754+0000 7fa5b19c4640 1 librados: starting objecter 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.754+0000 7fa5b19c4640 1 Processor -- start 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.754+0000 7fa5b19c4640 1 -- start start 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.754+0000 7fa5b19c4640 1 librados: setting wanted keys 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.754+0000 7fa5b19c4640 1 librados: calling monclient init 2024-03-29T10:51:49.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.754+0000 7fa5b19c4640 10 monclient: init 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.755+0000 7fa5b19c4640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.755+0000 7fa5b19c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.756+0000 7fa5b19c4640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ac0a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:49.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.756+0000 7fa5aaffd640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fa5ac0a9d70 0x7fa5ac1402e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:42504/0 (socket says 172.21.15.5:42504) 2024-03-29T10:51:49.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.756+0000 7fa5aaffd640 1 -- 172.21.15.5:0/1990602075 learned_addr learned my addr 172.21.15.5:0/1990602075 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:49.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.758+0000 7fa593fff640 10 client.?.objecter ms_handle_connect 0x7fa5ac0a9d70 2024-03-29T10:51:49.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.758+0000 7fa593fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:49.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.758+0000 7fa5b19c4640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ac13f650 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:49.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.758+0000 7fa5b19c4640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ac1457f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.758+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0ad7a0 con 0x7fa5ac1457f0 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.758+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac142d10 con 0x7fa5ac13f650 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.758+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ac0ad4c0 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.758+0000 7fa5b19c4640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.758+0000 7fa5b19c4640 10 monclient(hunting): authenticate will time out at 1122.999718s 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.760+0000 7fa593fff640 10 client.?.objecter ms_handle_connect 0x7fa5ac13f650 2024-03-29T10:51:49.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.760+0000 7fa593fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.760+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658520772 0 0) 0x7fa5ac0ad4c0 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:49.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa598004eb0 con 0x7fa5ac0a9d70 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 10 client.?.objecter ms_handle_connect 0x7fa5ac1457f0 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012411315 0 0) 0x7fa5ac142d10 con 0x7fa5ac13f650 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.761+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac0ad4c0 con 0x7fa5ac13f650 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857390646 0 0) 0x7fa5ac0ad7a0 con 0x7fa5ac1457f0 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:49.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac142d10 con 0x7fa5ac1457f0 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611149270 0 0) 0x7fa5ac142d10 con 0x7fa5ac1457f0 2024-03-29T10:51:49.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5ac0ad7a0 con 0x7fa5ac1457f0 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa5a4001d50 con 0x7fa5ac1457f0 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 10 monclient(hunting): dump: 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:49.749 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1337383860 0 0) 0x7fa5ac0ad7a0 con 0x7fa5ac1457f0 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.762+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 >> v1:172.21.15.5:6790/0 conn(0x7fa5ac0a9d70 legacy=0x7fa5ac1402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 >> v1:172.21.15.177:6789/0 conn(0x7fa5ac13f650 legacy=0x7fa5ac0ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 1 monclient: found mon.a 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:51:49.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac138b90 con 0x7fa5ac1457f0 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 10 monclient: _finish_auth 0 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5a40020b0 con 0x7fa5ac1457f0 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa593fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.763+0000 7fa5a97fa640 4 set_mon_vals no callback set 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.764+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa5a4005110 con 0x7fa5ac1457f0 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.764+0000 7fa593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.764+0000 7fa593fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.764+0000 7fa593fff640 10 monclient: dump: 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:49.751 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.764+0000 7fa5b19c4640 5 monclient: authenticate success, global_id 4356 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.765+0000 7fa5b19c4640 10 monclient: _renew_subs 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.765+0000 7fa5b19c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.765+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ac0ac420 con 0x7fa5ac1457f0 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.765+0000 7fa5b19c4640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.765+0000 7fa5b19c4640 10 monclient: _renew_subs 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.765+0000 7fa5b19c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.765+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ac1daa00 con 0x7fa5ac1457f0 2024-03-29T10:51:49.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.765+0000 7fa5b19c4640 1 librados: init done 2024-03-29T10:51:49.753 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.768+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fa5a401b0e0 con 0x7fa5ac1457f0 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.768+0000 7fa5b19c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.768+0000 7fa5b19c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.768+0000 7fa5b19c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.768+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ac142b00 con 0x7fa5ac1457f0 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.769+0000 7fa5aaffd640 10 monclient: get_auth_request con 0x7fa59806dd40 auth_method 0 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.769+0000 7fa593fff640 10 client.4356.objecter ms_handle_connect 0x7fa59806dd40 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.769+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fa5a4084db0 con 0x7fa5ac1457f0 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.769+0000 7fa593fff640 10 client.4356.objecter ms_dispatch 0x7fa5ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.769+0000 7fa593fff640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.769+0000 7fa593fff640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:49.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.770+0000 7fa593fff640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-29T10:51:49.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.779+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5a4050690 con 0x7fa5ac1457f0 2024-03-29T10:51:49.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.779+0000 7fa593fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:49.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:49.779+0000 7fa593fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:50.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.111+0000 7fa5b19c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:50.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.111+0000 7fa5b19c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:50.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.111+0000 7fa5b19c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:51:50.322 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.111+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5ac075f10 con 0x7fa5ac1457f0 2024-03-29T10:51:50.322 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.111+0000 7fa593fff640 1 -- 172.21.15.5:0/1990602075 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fa5a4001590 con 0x7fa5ac1457f0 2024-03-29T10:51:50.323 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.112+0000 7fa593fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:50.323 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.112+0000 7fa593fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:50.323 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:51:50.323 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:50.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.115+0000 7fa5b19c4640 10 librados: watch_flush enter 2024-03-29T10:51:50.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.116+0000 7fa5b19c4640 10 librados: watch_flush exit 2024-03-29T10:51:50.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.116+0000 7fa5b19c4640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-29T10:51:50.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.116+0000 7fa5b19c4640 10 client.4356.objecter successfully canceled tick 2024-03-29T10:51:50.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.116+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 >> v1:172.21.15.5:6816/2847500151 conn(0x7fa59806dd40 legacy=0x7fa598070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:50.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.116+0000 7fa5b19c4640 10 monclient: shutdown 2024-03-29T10:51:50.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.116+0000 7fa5b19c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:50.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.116+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 >> v1:172.21.15.5:6789/0 conn(0x7fa5ac1457f0 legacy=0x7fa5ac0abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:50.325 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.116+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 shutdown_connections 2024-03-29T10:51:50.326 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.116+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 >> 172.21.15.5:0/1990602075 conn(0x7fa5ac066a00 msgr2=0x7fa5ac0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:50.326 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.117+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 shutdown_connections 2024-03-29T10:51:50.326 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.117+0000 7fa5b19c4640 1 -- 172.21.15.5:0/1990602075 wait complete. 2024-03-29T10:51:50.326 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:50.117+0000 7fa5b19c4640 1 librados: shutdown 2024-03-29T10:51:50.326 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:57.133 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:57.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.300+0000 7fee3a815640 10 monclient: get_monmap_and_config 2024-03-29T10:51:57.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.300+0000 7fee3a815640 10 monclient: build_initial_monmap 2024-03-29T10:51:57.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.300+0000 7fee3a815640 10 monclient: monmap: 2024-03-29T10:51:57.285 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:57.285 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:57.285 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:51:57.302153+0000 2024-03-29T10:51:57.286 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:51:57.302153+0000 2024-03-29T10:51:57.286 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:57.286 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:57.286 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:57.287 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:51:57.287 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:51:57.287 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:57.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.304+0000 7fee3a815640 1 Processor -- start 2024-03-29T10:51:57.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.304+0000 7fee3a815640 1 -- start start 2024-03-29T10:51:57.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.304+0000 7fee3a815640 10 monclient: init 2024-03-29T10:51:57.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.305+0000 7fee3a815640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:57.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.305+0000 7fee3a815640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:57.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.305+0000 7fee3a815640 10 monclient(hunting): picked mon.noname-a con 0x7fee341422b0 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:57.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.305+0000 7fee3a815640 10 monclient(hunting): picked mon.noname-b con 0x7fee34142690 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:57.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.305+0000 7fee3a815640 10 monclient(hunting): picked mon.noname-c con 0x7fee340b17d0 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:57.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.305+0000 7fee3a815640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee34142c80 con 0x7fee341422b0 2024-03-29T10:51:57.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.305+0000 7fee3a815640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee340aae40 con 0x7fee340b17d0 2024-03-29T10:51:57.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.305+0000 7fee3a815640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3413fbd0 con 0x7fee34142690 2024-03-29T10:51:57.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.305+0000 7fee3a815640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:57.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.305+0000 7fee3a815640 10 monclient(hunting): authenticate will time out at 1130.547688s 2024-03-29T10:51:57.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.306+0000 7fee337fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fee34142690 0x7fee340b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:38304/0 (socket says 172.21.15.5:38304) 2024-03-29T10:51:57.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.306+0000 7fee337fe640 1 -- 172.21.15.5:0/2017337854 learned_addr learned my addr 172.21.15.5:0/2017337854 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:57.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.308+0000 7fee32ffd640 1 -- 172.21.15.5:0/2017337854 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047969295 0 0) 0x7fee3413fbd0 con 0x7fee34142690 2024-03-29T10:51:57.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.308+0000 7fee32ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:57.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.308+0000 7fee32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:57.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.308+0000 7fee32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:57.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.308+0000 7fee32ffd640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:51:57.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.309+0000 7fee32ffd640 1 -- 172.21.15.5:0/2017337854 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee1c003520 con 0x7fee34142690 2024-03-29T10:51:57.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.311+0000 7fee32ffd640 1 -- 172.21.15.5:0/2017337854 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1077537056 0 0) 0x7fee1c003520 con 0x7fee34142690 2024-03-29T10:51:57.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.311+0000 7fee32ffd640 1 -- 172.21.15.5:0/2017337854 >> v1:172.21.15.177:6789/0 conn(0x7fee340b17d0 legacy=0x7fee340b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:57.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.311+0000 7fee32ffd640 1 -- 172.21.15.5:0/2017337854 >> v1:172.21.15.5:6789/0 conn(0x7fee341422b0 legacy=0x7fee3413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:57.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.311+0000 7fee32ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:57.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.311+0000 7fee32ffd640 1 monclient: found mon.noname-b 2024-03-29T10:51:57.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.311+0000 7fee32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 1 -- 172.21.15.5:0/2017337854 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee34151940 con 0x7fee34142690 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 1 -- 172.21.15.5:0/2017337854 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fee28001690 con 0x7fee34142690 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 10 monclient: dump: 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:57.295 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:57.296 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:57.296 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:57.296 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:57.296 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:57.296 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:57.296 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:57.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee3a815640 5 monclient: authenticate success, global_id 4412 2024-03-29T10:51:57.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee3a815640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:57.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 1 -- 172.21.15.5:0/2017337854 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee280018b0 con 0x7fee34142690 2024-03-29T10:51:57.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:57.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee3a815640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:57.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee3a815640 4 set_mon_vals no callback set 2024-03-29T10:51:57.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 1 -- 172.21.15.5:0/2017337854 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fee28001c40 con 0x7fee34142690 2024-03-29T10:51:57.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:57.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:57.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.312+0000 7fee32ffd640 10 monclient: dump: 2024-03-29T10:51:57.298 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:57.298 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:57.298 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:57.298 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.313+0000 7fee3a815640 10 monclient: shutdown 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.313+0000 7fee3a815640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.313+0000 7fee3a815640 1 -- 172.21.15.5:0/2017337854 >> v1:172.21.15.5:6790/0 conn(0x7fee34142690 legacy=0x7fee340b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.313+0000 7fee3a815640 1 -- 172.21.15.5:0/2017337854 shutdown_connections 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.313+0000 7fee3a815640 1 -- 172.21.15.5:0/2017337854 >> 172.21.15.5:0/2017337854 conn(0x7fee34066a00 msgr2=0x7fee34066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:57.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.314+0000 7fee3a815640 1 -- 172.21.15.5:0/2017337854 shutdown_connections 2024-03-29T10:51:57.302 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.315+0000 7fee3a815640 1 -- 172.21.15.5:0/2017337854 wait complete. 2024-03-29T10:51:57.302 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.315+0000 7fee3a815640 10 monclient: build_initial_monmap 2024-03-29T10:51:57.302 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.315+0000 7fee3a815640 10 monclient: monmap: 2024-03-29T10:51:57.302 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:51:57.302 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:57.302 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:51:57.302 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:51:57.302 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:57.302 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.316+0000 7fee3a815640 1 librados: starting msgr at 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.316+0000 7fee3a815640 1 librados: starting objecter 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.316+0000 7fee3a815640 1 Processor -- start 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 1 -- start start 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 1 librados: setting wanted keys 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 1 librados: calling monclient init 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 10 monclient: init 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:57.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 10 monclient(hunting): picked mon.noname-c con 0x7fee340b17d0 addr v1:172.21.15.5:6790/0 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 10 monclient(hunting): picked mon.noname-a con 0x7fee341422b0 addr v1:172.21.15.5:6789/0 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 10 monclient(hunting): picked mon.noname-b con 0x7fee34142690 addr v1:172.21.15.177:6789/0 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3412f680 con 0x7fee341422b0 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee341329c0 con 0x7fee34142690 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3406e9f0 con 0x7fee340b17d0 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.317+0000 7fee3a815640 10 monclient(hunting): authenticate will time out at 1130.560012s 2024-03-29T10:51:57.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.319+0000 7fee33fff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fee340b17d0 0x7fee341322b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:41542/0 (socket says 172.21.15.5:41542) 2024-03-29T10:51:57.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.319+0000 7fee33fff640 1 -- 172.21.15.5:0/723786808 learned_addr learned my addr 172.21.15.5:0/723786808 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.319+0000 7fee30ff9640 10 client.?.objecter ms_handle_connect 0x7fee340b17d0 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.319+0000 7fee30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004837963 0 0) 0x7fee3406e9f0 con 0x7fee340b17d0 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee20004eb0 con 0x7fee340b17d0 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3252964873 0 0) 0x7fee20004eb0 con 0x7fee340b17d0 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee3406e9f0 con 0x7fee340b17d0 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fee2c001db0 con 0x7fee340b17d0 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:57.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:57.309 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:57.309 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:57.309 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:57.309 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:57.309 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:57.309 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:57.309 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:57.309 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:57.309 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:57.310 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:57.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.320+0000 7fee30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:57.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.321+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2828408996 0 0) 0x7fee3406e9f0 con 0x7fee340b17d0 2024-03-29T10:51:57.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.321+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 >> v1:172.21.15.177:6789/0 conn(0x7fee34142690 legacy=0x7fee3412ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:57.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.321+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 >> v1:172.21.15.5:6789/0 conn(0x7fee341422b0 legacy=0x7fee3412e860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:57.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:57.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 1 monclient: found mon.c 2024-03-29T10:51:57.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:57.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3406ecd0 con 0x7fee340b17d0 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee2c001f10 con 0x7fee340b17d0 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fee2c003de0 con 0x7fee340b17d0 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee30ff9640 10 monclient: dump: 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:57.312 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee3a815640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee3a815640 10 monclient: _renew_subs 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee3a815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee3a815640 1 -- 172.21.15.5:0/723786808 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3413b9d0 con 0x7fee340b17d0 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee3a815640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee3a815640 10 monclient: _renew_subs 2024-03-29T10:51:57.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee3a815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:57.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee3a815640 1 -- 172.21.15.5:0/723786808 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3412fba0 con 0x7fee340b17d0 2024-03-29T10:51:57.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.322+0000 7fee3a815640 1 librados: init done 2024-03-29T10:51:57.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.324+0000 7fee31ffb640 4 set_mon_vals no callback set 2024-03-29T10:51:57.315 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.325+0000 7fee3a815640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:57.315 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.325+0000 7fee3a815640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:57.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.325+0000 7fee3a815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:57.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.325+0000 7fee3a815640 1 -- 172.21.15.5:0/723786808 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee14005080 con 0x7fee340b17d0 2024-03-29T10:51:57.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.327+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fee2c003f40 con 0x7fee340b17d0 2024-03-29T10:51:57.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.328+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fee2c084f60 con 0x7fee340b17d0 2024-03-29T10:51:57.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.328+0000 7fee337fe640 10 monclient: get_auth_request con 0x7fee2006dbe0 auth_method 0 2024-03-29T10:51:57.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.329+0000 7fee30ff9640 10 client.4415.objecter ms_dispatch 0x7fee34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:57.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.329+0000 7fee30ff9640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:57.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.329+0000 7fee30ff9640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:57.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.329+0000 7fee30ff9640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T10:51:57.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.329+0000 7fee30ff9640 10 client.4415.objecter ms_handle_connect 0x7fee2006dbe0 2024-03-29T10:51:57.322 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.338+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee2c050840 con 0x7fee340b17d0 2024-03-29T10:51:57.322 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.338+0000 7fee30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:57.323 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.338+0000 7fee30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:57.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.669+0000 7fee3a815640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:57.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.669+0000 7fee3a815640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:57.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.669+0000 7fee3a815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:51:57.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.669+0000 7fee3a815640 1 -- 172.21.15.5:0/723786808 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee14005330 con 0x7fee340b17d0 2024-03-29T10:51:57.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.671+0000 7fee30ff9640 1 -- 172.21.15.5:0/723786808 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fee2c052770 con 0x7fee340b17d0 2024-03-29T10:51:57.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.671+0000 7fee30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:57.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.671+0000 7fee30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:57.653 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:51:57.653 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:57.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 10 librados: watch_flush enter 2024-03-29T10:51:57.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 10 librados: watch_flush exit 2024-03-29T10:51:57.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T10:51:57.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 10 client.4415.objecter successfully canceled tick 2024-03-29T10:51:57.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 1 -- 172.21.15.5:0/723786808 >> v1:172.21.15.5:6816/2847500151 conn(0x7fee2006dbe0 legacy=0x7fee200700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:57.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 10 monclient: shutdown 2024-03-29T10:51:57.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:57.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 1 -- 172.21.15.5:0/723786808 >> v1:172.21.15.5:6790/0 conn(0x7fee340b17d0 legacy=0x7fee341322b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:57.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 1 -- 172.21.15.5:0/723786808 shutdown_connections 2024-03-29T10:51:57.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 1 -- 172.21.15.5:0/723786808 >> 172.21.15.5:0/723786808 conn(0x7fee34066a00 msgr2=0x7fee340b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:57.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 1 -- 172.21.15.5:0/723786808 shutdown_connections 2024-03-29T10:51:57.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.675+0000 7fee3a815640 1 -- 172.21.15.5:0/723786808 wait complete. 2024-03-29T10:51:57.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:51:57.676+0000 7fee3a815640 1 librados: shutdown 2024-03-29T10:51:57.677 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:04.483 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:04.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.648+0000 7fea8d425640 10 monclient: get_monmap_and_config 2024-03-29T10:52:04.630 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.648+0000 7fea8d425640 10 monclient: build_initial_monmap 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.648+0000 7fea8d425640 10 monclient: monmap: 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:52:04.649962+0000 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:52:04.649962+0000 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:52:04.631 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:04.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.649+0000 7fea8d425640 1 Processor -- start 2024-03-29T10:52:04.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.649+0000 7fea8d425640 1 -- start start 2024-03-29T10:52:04.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.649+0000 7fea8d425640 10 monclient: init 2024-03-29T10:52:04.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.650+0000 7fea8d425640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:04.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.650+0000 7fea8d425640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:04.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.650+0000 7fea8d425640 10 monclient(hunting): picked mon.noname-c con 0x7fea881457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:04.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.650+0000 7fea8d425640 10 monclient(hunting): picked mon.noname-b con 0x7fea8813f650 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:04.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.650+0000 7fea8d425640 10 monclient(hunting): picked mon.noname-a con 0x7fea880a9d70 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:04.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.650+0000 7fea8d425640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea880aaef0 con 0x7fea880a9d70 2024-03-29T10:52:04.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.650+0000 7fea8d425640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea88145bd0 con 0x7fea881457f0 2024-03-29T10:52:04.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.650+0000 7fea8d425640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea88145eb0 con 0x7fea8813f650 2024-03-29T10:52:04.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.650+0000 7fea8d425640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:04.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.650+0000 7fea8d425640 10 monclient(hunting): authenticate will time out at 1137.891891s 2024-03-29T10:52:04.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.654+0000 7fea86ffd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fea881457f0 0x7fea8813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:33573/0 (socket says 172.21.15.5:33573) 2024-03-29T10:52:04.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.654+0000 7fea86ffd640 1 -- 172.21.15.5:0/895608466 learned_addr learned my addr 172.21.15.5:0/895608466 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:04.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209875059 0 0) 0x7fea880aaef0 con 0x7fea880a9d70 2024-03-29T10:52:04.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea70003520 con 0x7fea880a9d70 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931245918 0 0) 0x7fea88145bd0 con 0x7fea881457f0 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea880aaef0 con 0x7fea881457f0 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3492925806 0 0) 0x7fea70003520 con 0x7fea880a9d70 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 >> v1:172.21.15.5:6790/0 conn(0x7fea8813f650 legacy=0x7fea880b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 >> v1:172.21.15.177:6789/0 conn(0x7fea881457f0 legacy=0x7fea8813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 1 monclient: found mon.noname-a 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea88146340 con 0x7fea880a9d70 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fea78001690 con 0x7fea880a9d70 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.655+0000 7fea85ffb640 10 monclient: dump: 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:04.639 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:04.640 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.640 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:04.640 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:04.640 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:04.640 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:04.640 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:04.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.656+0000 7fea8d425640 5 monclient: authenticate success, global_id 4362 2024-03-29T10:52:04.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.656+0000 7fea8d425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:04.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.656+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea780017f0 con 0x7fea880a9d70 2024-03-29T10:52:04.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.656+0000 7fea85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:04.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.656+0000 7fea8d425640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:04.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.656+0000 7fea8d425640 4 set_mon_vals no callback set 2024-03-29T10:52:04.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.657+0000 7fea85ffb640 1 -- 172.21.15.5:0/895608466 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fea78001c00 con 0x7fea880a9d70 2024-03-29T10:52:04.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.657+0000 7fea85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.657+0000 7fea85ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.657+0000 7fea8d425640 10 monclient: shutdown 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.657+0000 7fea85ffb640 10 monclient: dump: 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:04.642 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:04.643 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:04.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.657+0000 7fea8d425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:04.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.657+0000 7fea8d425640 1 -- 172.21.15.5:0/895608466 >> v1:172.21.15.5:6789/0 conn(0x7fea880a9d70 legacy=0x7fea8814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.658+0000 7fea8d425640 1 -- 172.21.15.5:0/895608466 shutdown_connections 2024-03-29T10:52:04.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.658+0000 7fea8d425640 1 -- 172.21.15.5:0/895608466 >> 172.21.15.5:0/895608466 conn(0x7fea88066a00 msgr2=0x7fea88066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:04.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.659+0000 7fea8d425640 1 -- 172.21.15.5:0/895608466 shutdown_connections 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.659+0000 7fea8d425640 1 -- 172.21.15.5:0/895608466 wait complete. 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.660+0000 7fea8d425640 10 monclient: build_initial_monmap 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.660+0000 7fea8d425640 10 monclient: monmap: 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:52:04.645 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.660+0000 7fea8d425640 1 librados: starting msgr at 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.660+0000 7fea8d425640 1 librados: starting objecter 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.660+0000 7fea8d425640 1 Processor -- start 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 1 -- start start 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 1 librados: setting wanted keys 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 1 librados: calling monclient init 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 10 monclient: init 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 10 monclient(hunting): picked mon.noname-b con 0x7fea880a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 10 monclient(hunting): picked mon.noname-a con 0x7fea8813f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 10 monclient(hunting): picked mon.noname-c con 0x7fea881457f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:04.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea88143a80 con 0x7fea8813f650 2024-03-29T10:52:04.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea88144e20 con 0x7fea880a9d70 2024-03-29T10:52:04.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea88143560 con 0x7fea881457f0 2024-03-29T10:52:04.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:04.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.661+0000 7fea8d425640 10 monclient(hunting): authenticate will time out at 1137.903048s 2024-03-29T10:52:04.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.665+0000 7fea877fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fea881457f0 0x7fea881eec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:35023/0 (socket says 172.21.15.5:35023) 2024-03-29T10:52:04.647 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.665+0000 7fea877fe640 1 -- 172.21.15.5:0/4153351393 learned_addr learned my addr 172.21.15.5:0/4153351393 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 client.?.objecter ms_handle_connect 0x7fea881457f0 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 client.?.objecter ms_handle_connect 0x7fea8813f650 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124660193 0 0) 0x7fea88143a80 con 0x7fea8813f650 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 monclient(hunting): my global_id is 4365 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea74004eb0 con 0x7fea8813f650 2024-03-29T10:52:04.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038209803 0 0) 0x7fea88143560 con 0x7fea881457f0 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.668+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea88143a80 con 0x7fea881457f0 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2791572299 0 0) 0x7fea88143a80 con 0x7fea881457f0 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea88143560 con 0x7fea881457f0 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fea78001b00 con 0x7fea881457f0 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:04.657 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2642333635 0 0) 0x7fea74004eb0 con 0x7fea8813f650 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea88143a80 con 0x7fea8813f650 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fea7c0017a0 con 0x7fea8813f650 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:04.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3177973970 0 0) 0x7fea88143560 con 0x7fea881457f0 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 >> v1:172.21.15.177:6789/0 conn(0x7fea880a9d70 legacy=0x7fea881e4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 >> v1:172.21.15.5:6789/0 conn(0x7fea8813f650 legacy=0x7fea881e4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:04.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 monclient: found mon.c 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea881436f0 con 0x7fea881457f0 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea8d425640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea8d425640 10 monclient: _renew_subs 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea8d425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea8d425640 1 -- 172.21.15.5:0/4153351393 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8813e740 con 0x7fea881457f0 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea8d425640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea8d425640 10 monclient: _renew_subs 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea8d425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea8d425640 1 -- 172.21.15.5:0/4153351393 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea881391e0 con 0x7fea881457f0 2024-03-29T10:52:04.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea8d425640 1 librados: init done 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea780038a0 con 0x7fea881457f0 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.669+0000 7fea857fa640 4 set_mon_vals no callback set 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.670+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fea78003cf0 con 0x7fea881457f0 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.670+0000 7fea6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.670+0000 7fea6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.670+0000 7fea6f7fe640 10 monclient: dump: 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:04.661 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.671+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fea78003e50 con 0x7fea881457f0 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.672+0000 7fea8d425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.672+0000 7fea8d425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.672+0000 7fea8d425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.672+0000 7fea8d425640 1 -- 172.21.15.5:0/4153351393 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea68005080 con 0x7fea881457f0 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.672+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fea78084aa0 con 0x7fea881457f0 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.672+0000 7fea6f7fe640 10 client.4418.objecter ms_dispatch 0x7fea88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.672+0000 7fea6f7fe640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.672+0000 7fea6f7fe640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.673+0000 7fea6f7fe640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-29T10:52:04.662 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.673+0000 7fea86ffd640 10 monclient: get_auth_request con 0x7fea7406df10 auth_method 0 2024-03-29T10:52:04.663 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.675+0000 7fea6f7fe640 10 client.4418.objecter ms_handle_connect 0x7fea7406df10 2024-03-29T10:52:04.663 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.681+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea78050380 con 0x7fea881457f0 2024-03-29T10:52:04.663 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.681+0000 7fea6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:04.663 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:04.682+0000 7fea6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:05.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.020+0000 7fea8d425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:05.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.020+0000 7fea8d425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:05.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.020+0000 7fea8d425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:05.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.020+0000 7fea8d425640 1 -- 172.21.15.5:0/4153351393 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea68005330 con 0x7fea881457f0 2024-03-29T10:52:05.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.022+0000 7fea6f7fe640 1 -- 172.21.15.5:0/4153351393 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fea780522b0 con 0x7fea881457f0 2024-03-29T10:52:05.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.022+0000 7fea6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:05.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.022+0000 7fea6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:05.004 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:52:05.005 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:05.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.025+0000 7fea8d425640 10 librados: watch_flush enter 2024-03-29T10:52:05.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.025+0000 7fea8d425640 10 librados: watch_flush exit 2024-03-29T10:52:05.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.025+0000 7fea8d425640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-29T10:52:05.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.025+0000 7fea8d425640 10 client.4418.objecter successfully canceled tick 2024-03-29T10:52:05.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.026+0000 7fea8d425640 1 -- 172.21.15.5:0/4153351393 >> v1:172.21.15.5:6816/2847500151 conn(0x7fea7406df10 legacy=0x7fea740703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:05.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.026+0000 7fea8d425640 10 monclient: shutdown 2024-03-29T10:52:05.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.026+0000 7fea8d425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:05.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.026+0000 7fea8d425640 1 -- 172.21.15.5:0/4153351393 >> v1:172.21.15.5:6790/0 conn(0x7fea881457f0 legacy=0x7fea881eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:05.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.026+0000 7fea8d425640 1 -- 172.21.15.5:0/4153351393 shutdown_connections 2024-03-29T10:52:05.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.026+0000 7fea8d425640 1 -- 172.21.15.5:0/4153351393 >> 172.21.15.5:0/4153351393 conn(0x7fea88066a00 msgr2=0x7fea880b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:05.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.027+0000 7fea8d425640 1 -- 172.21.15.5:0/4153351393 shutdown_connections 2024-03-29T10:52:05.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.029+0000 7fea8d425640 1 -- 172.21.15.5:0/4153351393 wait complete. 2024-03-29T10:52:05.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:05.030+0000 7fea8d425640 1 librados: shutdown 2024-03-29T10:52:05.030 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:11.838 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:11.983 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.001+0000 7f556bfff640 10 monclient: get_monmap_and_config 2024-03-29T10:52:11.983 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.002+0000 7f556bfff640 10 monclient: build_initial_monmap 2024-03-29T10:52:11.984 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.002+0000 7f556bfff640 10 monclient: monmap: 2024-03-29T10:52:11.984 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:11.984 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:11.984 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:52:12.003026+0000 2024-03-29T10:52:11.984 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:52:12.003026+0000 2024-03-29T10:52:11.984 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:11.984 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:11.984 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:11.984 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:52:11.984 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:52:11.985 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:11.990 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.002+0000 7f556bfff640 1 Processor -- start 2024-03-29T10:52:11.990 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.002+0000 7f556bfff640 1 -- start start 2024-03-29T10:52:11.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.002+0000 7f556bfff640 10 monclient: init 2024-03-29T10:52:11.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:11.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:11.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f556c0b2800 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:11.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f556c0a9c00 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:11.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f556c0b38c0 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:11.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556bfff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c141eb0 con 0x7f556c0b2800 2024-03-29T10:52:11.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556bfff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c0b22d0 con 0x7f556c0b38c0 2024-03-29T10:52:11.991 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556bfff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c142d30 con 0x7f556c0a9c00 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556bfff640 10 monclient(hunting): authenticate will time out at 1145.244845s 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556affd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f556c0b2800 0x7f556c142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:37514/0 (socket says 172.21.15.5:37514) 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.003+0000 7f556affd640 1 -- 172.21.15.5:0/4077806159 learned_addr learned my addr 172.21.15.5:0/4077806159 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527638086 0 0) 0x7f556c141eb0 con 0x7f556c0b2800 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5558003520 con 0x7f556c0b2800 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151986920 0 0) 0x7f556c142d30 con 0x7f556c0a9c00 2024-03-29T10:52:11.992 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.004+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f556c141eb0 con 0x7f556c0a9c00 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1627713091 0 0) 0x7f5558003520 con 0x7f556c0b2800 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 >> v1:172.21.15.5:6790/0 conn(0x7f556c0a9c00 legacy=0x7f556c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 >> v1:172.21.15.177:6789/0 conn(0x7f556c0b38c0 legacy=0x7f556c14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 1 monclient: found mon.noname-a 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:11.993 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:52:11.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c0aaf90 con 0x7f556c0b2800 2024-03-29T10:52:11.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:11.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:11.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:11.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f556bfff640 5 monclient: authenticate success, global_id 4368 2024-03-29T10:52:11.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f556bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:11.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5564001a10 con 0x7f556c0b2800 2024-03-29T10:52:11.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:11.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:11.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: dump: 2024-03-29T10:52:11.995 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:11.995 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:11.996 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:11.996 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:11.996 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:11.996 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:11.996 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:11.997 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:11.997 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:11.997 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:11.997 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f556bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:11.997 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5564001bf0 con 0x7f556c0b2800 2024-03-29T10:52:11.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:11.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f556bfff640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:11.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f556bfff640 4 set_mon_vals no callback set 2024-03-29T10:52:11.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 1 -- 172.21.15.5:0/4077806159 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5564001a10 con 0x7f556c0b2800 2024-03-29T10:52:11.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:11.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:11.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.005+0000 7f5569ffb640 10 monclient: dump: 2024-03-29T10:52:11.999 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:11.999 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:11.999 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:11.999 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:11.999 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:11.999 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:12.000 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:12.000 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:12.000 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:12.000 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:12.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.006+0000 7f556bfff640 10 monclient: shutdown 2024-03-29T10:52:12.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.006+0000 7f556bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:12.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.006+0000 7f556bfff640 1 -- 172.21.15.5:0/4077806159 >> v1:172.21.15.5:6789/0 conn(0x7f556c0b2800 legacy=0x7f556c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.007+0000 7f556bfff640 1 -- 172.21.15.5:0/4077806159 shutdown_connections 2024-03-29T10:52:12.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.007+0000 7f556bfff640 1 -- 172.21.15.5:0/4077806159 >> 172.21.15.5:0/4077806159 conn(0x7f556c066a00 msgr2=0x7f556c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:12.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.007+0000 7f556bfff640 1 -- 172.21.15.5:0/4077806159 shutdown_connections 2024-03-29T10:52:12.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.007+0000 7f556bfff640 1 -- 172.21.15.5:0/4077806159 wait complete. 2024-03-29T10:52:12.001 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.007+0000 7f556bfff640 10 monclient: build_initial_monmap 2024-03-29T10:52:12.002 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.007+0000 7f556bfff640 10 monclient: monmap: 2024-03-29T10:52:12.002 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:12.002 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:12.002 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:52:12.002 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:52:12.002 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:12.003 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:12.003 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:12.003 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:52:12.003 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:52:12.003 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:12.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.008+0000 7f556bfff640 1 librados: starting msgr at 2024-03-29T10:52:12.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.008+0000 7f556bfff640 1 librados: starting objecter 2024-03-29T10:52:12.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.008+0000 7f556bfff640 1 Processor -- start 2024-03-29T10:52:12.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.008+0000 7f556bfff640 1 -- start start 2024-03-29T10:52:12.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.009+0000 7f556bfff640 1 librados: setting wanted keys 2024-03-29T10:52:12.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.009+0000 7f556bfff640 1 librados: calling monclient init 2024-03-29T10:52:12.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.009+0000 7f556bfff640 10 monclient: init 2024-03-29T10:52:12.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.009+0000 7f556bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:12.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.009+0000 7f556bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:12.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f556bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f556c0a9c00 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:12.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f556bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f556c0b2800 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:12.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f556bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f556c0b38c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:12.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f556bfff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c0aca00 con 0x7f556c0b38c0 2024-03-29T10:52:12.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f556bfff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c0acce0 con 0x7f556c0b2800 2024-03-29T10:52:12.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f556bfff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c1da330 con 0x7f556c0a9c00 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f556bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f556bfff640 10 monclient(hunting): authenticate will time out at 1145.251235s 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f556affd640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f556c0a9c00 0x7f556c13eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:54494/0 (socket says 172.21.15.5:54494) 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f556affd640 1 -- 172.21.15.5:0/3382809271 learned_addr learned my addr 172.21.15.5:0/3382809271 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f5563fff640 10 client.?.objecter ms_handle_connect 0x7f556c0a9c00 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f5563fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f5563fff640 10 client.?.objecter ms_handle_connect 0x7f556c0b38c0 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.010+0000 7f5563fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467917998 0 0) 0x7f556c1da330 con 0x7f556c0a9c00 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f554c004eb0 con 0x7f556c0a9c00 2024-03-29T10:52:12.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114718970 0 0) 0x7f556c0aca00 con 0x7f556c0b38c0 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f556c1da330 con 0x7f556c0b38c0 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 10 client.?.objecter ms_handle_connect 0x7f556c0b2800 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.011+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4250658932 0 0) 0x7f554c004eb0 con 0x7f556c0a9c00 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f556c0aca00 con 0x7f556c0a9c00 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943319925 0 0) 0x7f556c0acce0 con 0x7f556c0b2800 2024-03-29T10:52:12.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:12.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:12.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:12.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:52:12.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f554c004eb0 con 0x7f556c0b2800 2024-03-29T10:52:12.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5564001a10 con 0x7f556c0a9c00 2024-03-29T10:52:12.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:12.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:12.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 10 monclient(hunting): dump: 2024-03-29T10:52:12.015 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:12.015 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 299026353 0 0) 0x7f556c0aca00 con 0x7f556c0a9c00 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 >> v1:172.21.15.177:6789/0 conn(0x7f556c0b2800 legacy=0x7f556c1da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 >> v1:172.21.15.5:6789/0 conn(0x7f556c0b38c0 legacy=0x7f556c137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:12.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.012+0000 7f5563fff640 1 monclient: found mon.c 2024-03-29T10:52:12.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f5563fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:12.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f5563fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:12.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c13c480 con 0x7f556c0a9c00 2024-03-29T10:52:12.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f5563fff640 10 monclient: _finish_auth 0 2024-03-29T10:52:12.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f5563fff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:12.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f5563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:12.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f556bfff640 5 monclient: authenticate success, global_id 4424 2024-03-29T10:52:12.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f556bfff640 10 monclient: _renew_subs 2024-03-29T10:52:12.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5564004ec0 con 0x7f556c0a9c00 2024-03-29T10:52:12.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f556bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f556bfff640 1 -- 172.21.15.5:0/3382809271 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556c1daf60 con 0x7f556c0a9c00 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f556bfff640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f556bfff640 10 monclient: _renew_subs 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f556bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f556bfff640 1 -- 172.21.15.5:0/3382809271 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c0ab7a0 con 0x7f556c0a9c00 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f556bfff640 1 librados: init done 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f5563fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.013+0000 7f55697fa640 4 set_mon_vals no callback set 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.014+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5564005290 con 0x7f556c0a9c00 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.014+0000 7f5563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.015+0000 7f5563fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:12.018 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.015+0000 7f5563fff640 10 monclient: dump: 2024-03-29T10:52:12.019 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:12.019 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:12.019 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:12.019 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:12.019 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:12.019 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:12.019 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:12.019 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:12.019 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:12.019 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.015+0000 7f5561ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.015+0000 7f5561ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.015+0000 7f5561ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.015+0000 7f5561ffb640 1 -- 172.21.15.5:0/3382809271 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c1da4c0 con 0x7f556c0a9c00 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.016+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f556401b260 con 0x7f556c0a9c00 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.017+0000 7f556a7fc640 10 monclient: get_auth_request con 0x7f554c06dc00 auth_method 0 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.017+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f5564084fb0 con 0x7f556c0a9c00 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.017+0000 7f5563fff640 10 client.4424.objecter ms_dispatch 0x7f556c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.017+0000 7f5563fff640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.017+0000 7f5563fff640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.017+0000 7f5563fff640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T10:52:12.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.017+0000 7f5563fff640 10 client.4424.objecter ms_handle_connect 0x7f554c06dc00 2024-03-29T10:52:12.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.025+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55640508f0 con 0x7f556c0a9c00 2024-03-29T10:52:12.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.025+0000 7f5563fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:12.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.025+0000 7f5563fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:12.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.393+0000 7f5561ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:12.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.393+0000 7f5561ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:12.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.393+0000 7f5561ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:12.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.393+0000 7f5561ffb640 1 -- 172.21.15.5:0/3382809271 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f556c1db0f0 con 0x7f556c0a9c00 2024-03-29T10:52:12.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.394+0000 7f5563fff640 1 -- 172.21.15.5:0/3382809271 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f5564052820 con 0x7f556c0a9c00 2024-03-29T10:52:12.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.394+0000 7f5563fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:12.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.394+0000 7f5563fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:12.376 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:52:12.377 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:12.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.398+0000 7f5561ffb640 10 librados: watch_flush enter 2024-03-29T10:52:12.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.398+0000 7f5561ffb640 10 librados: watch_flush exit 2024-03-29T10:52:12.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.398+0000 7f5561ffb640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T10:52:12.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.398+0000 7f5561ffb640 10 client.4424.objecter successfully canceled tick 2024-03-29T10:52:12.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.398+0000 7f5561ffb640 1 -- 172.21.15.5:0/3382809271 >> v1:172.21.15.5:6816/2847500151 conn(0x7f554c06dc00 legacy=0x7f554c0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.398+0000 7f5561ffb640 10 monclient: shutdown 2024-03-29T10:52:12.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.398+0000 7f5561ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:12.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.398+0000 7f5561ffb640 1 -- 172.21.15.5:0/3382809271 >> v1:172.21.15.5:6790/0 conn(0x7f556c0a9c00 legacy=0x7f556c13eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:12.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.398+0000 7f5561ffb640 1 -- 172.21.15.5:0/3382809271 shutdown_connections 2024-03-29T10:52:12.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.398+0000 7f5561ffb640 1 -- 172.21.15.5:0/3382809271 >> 172.21.15.5:0/3382809271 conn(0x7f556c066a00 msgr2=0x7f556c13bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:12.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.399+0000 7f5561ffb640 1 -- 172.21.15.5:0/3382809271 shutdown_connections 2024-03-29T10:52:12.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.399+0000 7f5561ffb640 1 -- 172.21.15.5:0/3382809271 wait complete. 2024-03-29T10:52:12.381 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:12.399+0000 7f5561ffb640 1 librados: shutdown 2024-03-29T10:52:12.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:19.204 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:19.353 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.372+0000 7f5abba63640 10 monclient: get_monmap_and_config 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.372+0000 7f5abba63640 10 monclient: build_initial_monmap 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.372+0000 7f5abba63640 10 monclient: monmap: 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:52:19.373061+0000 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:52:19.373061+0000 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:52:19.354 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:19.355 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.373+0000 7f5abba63640 1 Processor -- start 2024-03-29T10:52:19.355 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.374+0000 7f5abba63640 1 -- start start 2024-03-29T10:52:19.355 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.374+0000 7f5abba63640 10 monclient: init 2024-03-29T10:52:19.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.375+0000 7f5abba63640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:19.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.375+0000 7f5abba63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:19.356 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.375+0000 7f5abba63640 10 monclient(hunting): picked mon.noname-a con 0x7f5ab41422b0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:19.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.375+0000 7f5abba63640 10 monclient(hunting): picked mon.noname-b con 0x7f5ab4142690 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:19.357 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.375+0000 7f5abba63640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab40b17d0 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:19.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.375+0000 7f5ab8ffe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f5ab4142690 0x7f5ab40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:45518/0 (socket says 172.21.15.5:45518) 2024-03-29T10:52:19.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.375+0000 7f5ab8ffe640 1 -- 172.21.15.5:0/1001077070 learned_addr learned my addr 172.21.15.5:0/1001077070 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:19.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.376+0000 7f5abba63640 1 -- 172.21.15.5:0/1001077070 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab4142c80 con 0x7f5ab41422b0 2024-03-29T10:52:19.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.376+0000 7f5abba63640 1 -- 172.21.15.5:0/1001077070 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab40aae40 con 0x7f5ab40b17d0 2024-03-29T10:52:19.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.376+0000 7f5abba63640 1 -- 172.21.15.5:0/1001077070 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab413fbd0 con 0x7f5ab4142690 2024-03-29T10:52:19.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.376+0000 7f5abba63640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:19.358 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.376+0000 7f5abba63640 10 monclient(hunting): authenticate will time out at 1152.617479s 2024-03-29T10:52:19.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269436313 0 0) 0x7f5ab4142c80 con 0x7f5ab41422b0 2024-03-29T10:52:19.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 10 monclient(hunting): my global_id is 4374 2024-03-29T10:52:19.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa0003520 con 0x7f5ab41422b0 2024-03-29T10:52:19.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005784939 0 0) 0x7f5ab413fbd0 con 0x7f5ab4142690 2024-03-29T10:52:19.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:52:19.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.378+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab4142c80 con 0x7f5ab4142690 2024-03-29T10:52:19.366 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.379+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 527412222 0 0) 0x7f5aa0003520 con 0x7f5ab41422b0 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.379+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 >> v1:172.21.15.5:6790/0 conn(0x7f5ab4142690 legacy=0x7f5ab40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.379+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 >> v1:172.21.15.177:6789/0 conn(0x7f5ab40b17d0 legacy=0x7f5ab40b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 1 monclient: found mon.noname-a 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab4150930 con 0x7f5ab41422b0 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 10 monclient: _finish_auth 0 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:19.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5ab0001a10 con 0x7f5ab41422b0 2024-03-29T10:52:19.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:19.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.380+0000 7f5aabfff640 10 monclient: dump: 2024-03-29T10:52:19.368 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:19.368 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:19.368 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:19.368 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:19.368 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.368 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:19.368 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.381+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ab0001b70 con 0x7f5ab41422b0 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.381+0000 7f5aabfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.381+0000 7f5aabfff640 1 -- 172.21.15.5:0/1001077070 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5ab0001f00 con 0x7f5ab41422b0 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.381+0000 7f5aabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.381+0000 7f5aabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.381+0000 7f5aabfff640 10 monclient: dump: 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:19.369 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.383+0000 7f5abba63640 5 monclient: authenticate success, global_id 4374 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.383+0000 7f5abba63640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.383+0000 7f5abba63640 4 set_mon_vals no callback set 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.384+0000 7f5abba63640 10 monclient: shutdown 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.384+0000 7f5abba63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:19.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.384+0000 7f5abba63640 1 -- 172.21.15.5:0/1001077070 >> v1:172.21.15.5:6789/0 conn(0x7f5ab41422b0 legacy=0x7f5ab413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.384+0000 7f5abba63640 1 -- 172.21.15.5:0/1001077070 shutdown_connections 2024-03-29T10:52:19.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.384+0000 7f5abba63640 1 -- 172.21.15.5:0/1001077070 >> 172.21.15.5:0/1001077070 conn(0x7f5ab4066a00 msgr2=0x7f5ab4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:19.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.385+0000 7f5abba63640 1 -- 172.21.15.5:0/1001077070 shutdown_connections 2024-03-29T10:52:19.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.385+0000 7f5abba63640 1 -- 172.21.15.5:0/1001077070 wait complete. 2024-03-29T10:52:19.373 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.386+0000 7f5abba63640 10 monclient: build_initial_monmap 2024-03-29T10:52:19.373 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.386+0000 7f5abba63640 10 monclient: monmap: 2024-03-29T10:52:19.373 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:19.373 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:19.373 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:52:19.373 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:52:19.373 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:19.373 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:19.373 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:19.373 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.386+0000 7f5abba63640 1 librados: starting msgr at 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.386+0000 7f5abba63640 1 librados: starting objecter 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.386+0000 7f5abba63640 1 Processor -- start 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.386+0000 7f5abba63640 1 -- start start 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.387+0000 7f5abba63640 1 librados: setting wanted keys 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.387+0000 7f5abba63640 1 librados: calling monclient init 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.387+0000 7f5abba63640 10 monclient: init 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.388+0000 7f5abba63640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.388+0000 7f5abba63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:19.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.388+0000 7f5abba63640 10 monclient(hunting): picked mon.noname-a con 0x7f5ab40b17d0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:19.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.388+0000 7f5abba63640 10 monclient(hunting): picked mon.noname-b con 0x7f5ab41422b0 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:19.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.388+0000 7f5abba63640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab4142690 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:19.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.388+0000 7f5abba63640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab40b67f0 con 0x7f5ab40b17d0 2024-03-29T10:52:19.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.388+0000 7f5abba63640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab40aeb40 con 0x7f5ab41422b0 2024-03-29T10:52:19.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.388+0000 7f5abba63640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab40ada50 con 0x7f5ab4142690 2024-03-29T10:52:19.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.388+0000 7f5abba63640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:19.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.388+0000 7f5abba63640 10 monclient(hunting): authenticate will time out at 1152.629549s 2024-03-29T10:52:19.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.392+0000 7f5aba000640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f5ab4142690 0x7f5ab41e1c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:46500/0 (socket says 172.21.15.5:46500) 2024-03-29T10:52:19.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.392+0000 7f5aba000640 1 -- 172.21.15.5:0/946901401 learned_addr learned my addr 172.21.15.5:0/946901401 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:19.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.392+0000 7f5aa9ffb640 10 client.?.objecter ms_handle_connect 0x7f5ab4142690 2024-03-29T10:52:19.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.392+0000 7f5aa9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:19.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.392+0000 7f5aa9ffb640 10 client.?.objecter ms_handle_connect 0x7f5ab40b17d0 2024-03-29T10:52:19.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.392+0000 7f5aa9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:19.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.392+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259167567 0 0) 0x7f5ab40ada50 con 0x7f5ab4142690 2024-03-29T10:52:19.383 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:52:19.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a98004eb0 con 0x7f5ab4142690 2024-03-29T10:52:19.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 client.?.objecter ms_handle_connect 0x7f5ab41422b0 2024-03-29T10:52:19.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:19.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093688063 0 0) 0x7f5ab40aeb40 con 0x7f5ab41422b0 2024-03-29T10:52:19.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:19.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:19.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab40ada50 con 0x7f5ab41422b0 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3984641508 0 0) 0x7f5a98004eb0 con 0x7f5ab4142690 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ab40aeb40 con 0x7f5ab4142690 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5aac004000 con 0x7f5ab4142690 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:19.385 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:19.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 893074515 0 0) 0x7f5ab40ada50 con 0x7f5ab41422b0 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a98004eb0 con 0x7f5ab41422b0 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5aa4001770 con 0x7f5ab41422b0 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.393+0000 7f5aa9ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:19.387 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4216177166 0 0) 0x7f5ab40aeb40 con 0x7f5ab4142690 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 >> v1:172.21.15.177:6789/0 conn(0x7f5ab41422b0 legacy=0x7f5ab40ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 >> v1:172.21.15.5:6789/0 conn(0x7f5ab40b17d0 legacy=0x7f5ab40aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 1 monclient: found mon.c 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab40add30 con 0x7f5ab4142690 2024-03-29T10:52:19.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5aa9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5abba63640 5 monclient: authenticate success, global_id 4430 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5abba63640 10 monclient: _renew_subs 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5abba63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.394+0000 7f5abba63640 1 -- 172.21.15.5:0/946901401 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab40ae010 con 0x7f5ab4142690 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aac004160 con 0x7f5ab4142690 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5aa9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5aac0010b0 con 0x7f5ab4142690 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5aa9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5aa9ffb640 10 monclient: dump: 2024-03-29T10:52:19.389 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:19.390 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:19.390 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:19.390 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:19.390 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:19.390 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:19.390 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:19.390 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5aab7fe640 4 set_mon_vals no callback set 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5abba63640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5abba63640 10 monclient: _renew_subs 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5abba63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.395+0000 7f5abba63640 1 -- 172.21.15.5:0/946901401 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab40b6520 con 0x7f5ab4142690 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.396+0000 7f5abba63640 1 librados: init done 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.398+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f5aac001210 con 0x7f5ab4142690 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.399+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f5aac0847d0 con 0x7f5ab4142690 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.399+0000 7f5aa9ffb640 10 client.4430.objecter ms_dispatch 0x7f5ab4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:19.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.399+0000 7f5aa9ffb640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:19.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.399+0000 7f5aa9ffb640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:19.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.399+0000 7f5aa9ffb640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T10:52:19.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.399+0000 7f5ab97ff640 10 monclient: get_auth_request con 0x7f5a9806de00 auth_method 0 2024-03-29T10:52:19.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.400+0000 7f5abba63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:19.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.400+0000 7f5abba63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:19.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.400+0000 7f5abba63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:19.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.400+0000 7f5abba63640 1 -- 172.21.15.5:0/946901401 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab4075f10 con 0x7f5ab4142690 2024-03-29T10:52:19.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.401+0000 7f5aa9ffb640 10 client.4430.objecter ms_handle_connect 0x7f5a9806de00 2024-03-29T10:52:19.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.410+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5aac04fee0 con 0x7f5ab4142690 2024-03-29T10:52:19.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.410+0000 7f5aa9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:19.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.410+0000 7f5aa9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:19.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.760+0000 7f5abba63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:19.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.760+0000 7f5abba63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:19.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.760+0000 7f5abba63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:19.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.760+0000 7f5abba63640 1 -- 172.21.15.5:0/946901401 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ab40b2270 con 0x7f5ab4142690 2024-03-29T10:52:19.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.764+0000 7f5aa9ffb640 1 -- 172.21.15.5:0/946901401 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f5aac051fe0 con 0x7f5ab4142690 2024-03-29T10:52:19.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.764+0000 7f5aa9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:19.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.764+0000 7f5aa9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:19.746 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:52:19.746 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:19.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.768+0000 7f5abba63640 10 librados: watch_flush enter 2024-03-29T10:52:19.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.768+0000 7f5abba63640 10 librados: watch_flush exit 2024-03-29T10:52:19.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.768+0000 7f5abba63640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T10:52:19.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.768+0000 7f5abba63640 10 client.4430.objecter successfully canceled tick 2024-03-29T10:52:19.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.769+0000 7f5abba63640 1 -- 172.21.15.5:0/946901401 >> v1:172.21.15.5:6816/2847500151 conn(0x7f5a9806de00 legacy=0x7f5a980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.769+0000 7f5abba63640 10 monclient: shutdown 2024-03-29T10:52:19.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.769+0000 7f5abba63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:19.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.769+0000 7f5abba63640 1 -- 172.21.15.5:0/946901401 >> v1:172.21.15.5:6790/0 conn(0x7f5ab4142690 legacy=0x7f5ab41e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:19.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.769+0000 7f5abba63640 1 -- 172.21.15.5:0/946901401 shutdown_connections 2024-03-29T10:52:19.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.769+0000 7f5abba63640 1 -- 172.21.15.5:0/946901401 >> 172.21.15.5:0/946901401 conn(0x7f5ab4066a00 msgr2=0x7f5ab414f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:19.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.769+0000 7f5abba63640 1 -- 172.21.15.5:0/946901401 shutdown_connections 2024-03-29T10:52:19.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.769+0000 7f5abba63640 1 -- 172.21.15.5:0/946901401 wait complete. 2024-03-29T10:52:19.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:19.770+0000 7f5abba63640 1 librados: shutdown 2024-03-29T10:52:19.767 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:26.575 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:26.718 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.736+0000 7f8252dd1640 10 monclient: get_monmap_and_config 2024-03-29T10:52:26.718 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.736+0000 7f8252dd1640 10 monclient: build_initial_monmap 2024-03-29T10:52:26.718 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.736+0000 7f8252dd1640 10 monclient: monmap: 2024-03-29T10:52:26.718 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:26.718 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:26.718 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:52:26.737894+0000 2024-03-29T10:52:26.718 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:52:26.737894+0000 2024-03-29T10:52:26.719 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:26.719 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:26.719 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:26.719 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:52:26.719 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:52:26.719 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:26.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.737+0000 7f8252dd1640 1 Processor -- start 2024-03-29T10:52:26.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.738+0000 7f8252dd1640 1 -- start start 2024-03-29T10:52:26.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.738+0000 7f8252dd1640 10 monclient: init 2024-03-29T10:52:26.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.738+0000 7f8252dd1640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:26.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.738+0000 7f8252dd1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:26.722 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.738+0000 7f8252dd1640 10 monclient(hunting): picked mon.noname-b con 0x7f824c0b2800 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.738+0000 7f8252dd1640 10 monclient(hunting): picked mon.noname-c con 0x7f824c0a9c00 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.738+0000 7f8252dd1640 10 monclient(hunting): picked mon.noname-a con 0x7f824c0b38c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.738+0000 7f8252dd1640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c141eb0 con 0x7f824c0b38c0 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.739+0000 7f8252dd1640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c0b22d0 con 0x7f824c0a9c00 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.739+0000 7f8252dd1640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c142d30 con 0x7f824c0b2800 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.739+0000 7f8252dd1640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.739+0000 7f8252dd1640 10 monclient(hunting): authenticate will time out at 1159.980124s 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.739+0000 7f82525d0640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f824c0b38c0 0x7f824c14d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43828/0 (socket says 172.21.15.5:43828) 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.739+0000 7f82525d0640 1 -- 172.21.15.5:0/1676193060 learned_addr learned my addr 172.21.15.5:0/1676193060 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.739+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999050472 0 0) 0x7f824c142d30 con 0x7f824c0b2800 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient(hunting): my global_id is 4433 2024-03-29T10:52:26.723 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8238003520 con 0x7f824c0b2800 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917856713 0 0) 0x7f824c141eb0 con 0x7f824c0b38c0 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f824c142d30 con 0x7f824c0b38c0 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3485730140 0 0) 0x7f8238003520 con 0x7f824c0b2800 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 >> v1:172.21.15.177:6789/0 conn(0x7f824c0a9c00 legacy=0x7f824c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 >> v1:172.21.15.5:6789/0 conn(0x7f824c0b38c0 legacy=0x7f824c14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 monclient: found mon.noname-b 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c0aaf90 con 0x7f824c0b2800 2024-03-29T10:52:26.724 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient: _finish_auth 0 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8248001690 con 0x7f824c0b2800 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8252dd1640 5 monclient: authenticate success, global_id 4433 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8252dd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient: dump: 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:26.725 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82480018b0 con 0x7f824c0b2800 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 1 -- 172.21.15.5:0/1676193060 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8248001bc0 con 0x7f824c0b2800 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.740+0000 7f8250dcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.741+0000 7f8250dcd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.741+0000 7f8250dcd640 10 monclient: dump: 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.726 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:26.727 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:26.727 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:26.727 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:26.727 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:26.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.741+0000 7f8252dd1640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:26.727 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.741+0000 7f8252dd1640 4 set_mon_vals no callback set 2024-03-29T10:52:26.734 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.742+0000 7f8252dd1640 10 monclient: shutdown 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.742+0000 7f8252dd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.742+0000 7f8252dd1640 1 -- 172.21.15.5:0/1676193060 >> v1:172.21.15.5:6790/0 conn(0x7f824c0b2800 legacy=0x7f824c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.742+0000 7f8252dd1640 1 -- 172.21.15.5:0/1676193060 shutdown_connections 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.742+0000 7f8252dd1640 1 -- 172.21.15.5:0/1676193060 >> 172.21.15.5:0/1676193060 conn(0x7f824c066a00 msgr2=0x7f824c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.742+0000 7f8252dd1640 1 -- 172.21.15.5:0/1676193060 shutdown_connections 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.742+0000 7f8252dd1640 1 -- 172.21.15.5:0/1676193060 wait complete. 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.743+0000 7f8252dd1640 10 monclient: build_initial_monmap 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.743+0000 7f8252dd1640 10 monclient: monmap: 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:26.735 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.743+0000 7f8252dd1640 1 librados: starting msgr at 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.743+0000 7f8252dd1640 1 librados: starting objecter 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.744+0000 7f8252dd1640 1 Processor -- start 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.744+0000 7f8252dd1640 1 -- start start 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.744+0000 7f8252dd1640 1 librados: setting wanted keys 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.744+0000 7f8252dd1640 1 librados: calling monclient init 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.744+0000 7f8252dd1640 10 monclient: init 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f8252dd1640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f8252dd1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f8252dd1640 10 monclient(hunting): picked mon.noname-b con 0x7f824c0a9c00 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f8252dd1640 10 monclient(hunting): picked mon.noname-c con 0x7f824c0b2800 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:26.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f8252dd1640 10 monclient(hunting): picked mon.noname-a con 0x7f824c0b38c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f8252dd1640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c13c480 con 0x7f824c0b38c0 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f8252dd1640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c138710 con 0x7f824c0a9c00 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f82525d0640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f824c0b38c0 0x7f824c138000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45074/0 (socket says 172.21.15.5:45074) 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f82525d0640 1 -- 172.21.15.5:0/1875056898 learned_addr learned my addr 172.21.15.5:0/1875056898 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c0aca40 con 0x7f824c0b2800 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f8252dd1640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f8252dd1640 10 monclient(hunting): authenticate will time out at 1159.986501s 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f82427fc640 10 client.?.objecter ms_handle_connect 0x7f824c0b38c0 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f82427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f82427fc640 10 client.?.objecter ms_handle_connect 0x7f824c0b2800 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.745+0000 7f82427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922399478 0 0) 0x7f824c0aca40 con 0x7f824c0b2800 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8230004eb0 con 0x7f824c0b2800 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752944101 0 0) 0x7f824c13c480 con 0x7f824c0b38c0 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f824c0aca40 con 0x7f824c0b38c0 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 client.?.objecter ms_handle_connect 0x7f824c0a9c00 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155995421 0 0) 0x7f824c138710 con 0x7f824c0a9c00 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:26.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.746+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f824c13c480 con 0x7f824c0a9c00 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2184378702 0 0) 0x7f8230004eb0 con 0x7f824c0b2800 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f824c138710 con 0x7f824c0b2800 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f823c001870 con 0x7f824c0b2800 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient(hunting): dump: 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:26.739 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3570423003 0 0) 0x7f824c0aca40 con 0x7f824c0b38c0 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8230006ae0 con 0x7f824c0b38c0 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8244001b90 con 0x7f824c0b38c0 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient(hunting): dump: 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.740 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1846455264 0 0) 0x7f824c13c480 con 0x7f824c0a9c00 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f824c0aca40 con 0x7f824c0a9c00 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8248001550 con 0x7f824c0a9c00 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient(hunting): dump: 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.741 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 406620537 0 0) 0x7f824c138710 con 0x7f824c0b2800 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 >> v1:172.21.15.177:6789/0 conn(0x7f824c0a9c00 legacy=0x7f824c1da630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 >> v1:172.21.15.5:6789/0 conn(0x7f824c0b38c0 legacy=0x7f824c138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 monclient: found mon.c 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:26.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c0acd20 con 0x7f824c0b2800 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f82427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f8252dd1640 5 monclient: authenticate success, global_id 4436 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f8252dd1640 10 monclient: _renew_subs 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f8252dd1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824c0abc80 con 0x7f824c0b2800 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f8252dd1640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f8252dd1640 10 monclient: _renew_subs 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f8252dd1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f824c0abaa0 con 0x7f824c0b2800 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.747+0000 7f8252dd1640 1 librados: init done 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.748+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f823c0035f0 con 0x7f824c0b2800 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.748+0000 7f82427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:26.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.748+0000 7f82437fe640 4 set_mon_vals no callback set 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.749+0000 7f8252dd1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.749+0000 7f8252dd1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.749+0000 7f8252dd1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.749+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8234005080 con 0x7f824c0b2800 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.749+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f823c001870 con 0x7f824c0b2800 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.749+0000 7f82427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.749+0000 7f82427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.749+0000 7f82427fc640 10 monclient: dump: 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:26.744 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.750+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f823c0195d0 con 0x7f824c0b2800 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.751+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f823c083120 con 0x7f824c0b2800 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.751+0000 7f82427fc640 10 client.4436.objecter ms_dispatch 0x7f824c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.751+0000 7f82427fc640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.751+0000 7f82427fc640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.751+0000 7f82427fc640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.752+0000 7f8251dcf640 10 monclient: get_auth_request con 0x7f823006e4c0 auth_method 0 2024-03-29T10:52:26.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.753+0000 7f82427fc640 10 client.4436.objecter ms_handle_connect 0x7f823006e4c0 2024-03-29T10:52:26.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.758+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f823c04eb40 con 0x7f824c0b2800 2024-03-29T10:52:26.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.758+0000 7f82427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:26.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:26.758+0000 7f82427fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:27.142 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.160+0000 7f8252dd1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:27.142 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.160+0000 7f8252dd1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:27.142 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.160+0000 7f8252dd1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:27.142 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.160+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8234005330 con 0x7f824c0b2800 2024-03-29T10:52:27.143 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.162+0000 7f82427fc640 1 -- 172.21.15.5:0/1875056898 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f823c050a70 con 0x7f824c0b2800 2024-03-29T10:52:27.143 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.162+0000 7f82427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:27.143 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.162+0000 7f82427fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:27.143 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:52:27.144 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:27.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.165+0000 7f8252dd1640 10 librados: watch_flush enter 2024-03-29T10:52:27.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.165+0000 7f8252dd1640 10 librados: watch_flush exit 2024-03-29T10:52:27.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.165+0000 7f8252dd1640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T10:52:27.147 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.165+0000 7f8252dd1640 10 client.4436.objecter successfully canceled tick 2024-03-29T10:52:27.147 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.165+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 >> v1:172.21.15.5:6816/2847500151 conn(0x7f823006e4c0 legacy=0x7f8230070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:27.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.165+0000 7f8252dd1640 10 monclient: shutdown 2024-03-29T10:52:27.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.165+0000 7f8252dd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:27.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.165+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 >> v1:172.21.15.5:6790/0 conn(0x7f824c0b2800 legacy=0x7f824c1378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:27.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.166+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 shutdown_connections 2024-03-29T10:52:27.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.166+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 >> 172.21.15.5:0/1875056898 conn(0x7f824c066a00 msgr2=0x7f824c141960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:27.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.166+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 shutdown_connections 2024-03-29T10:52:27.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.166+0000 7f8252dd1640 1 -- 172.21.15.5:0/1875056898 wait complete. 2024-03-29T10:52:27.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:27.166+0000 7f8252dd1640 1 librados: shutdown 2024-03-29T10:52:27.162 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:33.970 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:34.105 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.124+0000 7f503ef6d640 10 monclient: get_monmap_and_config 2024-03-29T10:52:34.106 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.124+0000 7f503ef6d640 10 monclient: build_initial_monmap 2024-03-29T10:52:34.106 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.124+0000 7f503ef6d640 10 monclient: monmap: 2024-03-29T10:52:34.106 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:34.106 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:34.106 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:52:34.125468+0000 2024-03-29T10:52:34.106 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:52:34.125468+0000 2024-03-29T10:52:34.106 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:34.106 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:34.106 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:34.106 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:52:34.107 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:52:34.107 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:34.107 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.125+0000 7f503ef6d640 1 Processor -- start 2024-03-29T10:52:34.107 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.125+0000 7f503ef6d640 1 -- start start 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.125+0000 7f503ef6d640 10 monclient: init 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.126+0000 7f503ef6d640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.126+0000 7f503ef6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.126+0000 7f503ef6d640 10 monclient(hunting): picked mon.noname-c con 0x7f5038071de0 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.126+0000 7f503ef6d640 10 monclient(hunting): picked mon.noname-b con 0x7f50380721c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.126+0000 7f503ef6d640 10 monclient(hunting): picked mon.noname-a con 0x7f5038145ae0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.126+0000 7f503ef6d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5038145650 con 0x7f5038145ae0 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.126+0000 7f503ef6d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50380ae260 con 0x7f5038071de0 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.126+0000 7f503ef6d640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50381420d0 con 0x7f50380721c0 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.126+0000 7f503ef6d640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:34.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.126+0000 7f503ef6d640 10 monclient(hunting): authenticate will time out at 1167.367848s 2024-03-29T10:52:34.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f5037fff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f50380721c0 0x7f503813f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:50240/0 (socket says 172.21.15.5:50240) 2024-03-29T10:52:34.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f5037fff640 1 -- 172.21.15.5:0/1004071525 learned_addr learned my addr 172.21.15.5:0/1004071525 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:34.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f50377fe640 1 -- 172.21.15.5:0/1004071525 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072723780 0 0) 0x7f50381420d0 con 0x7f50380721c0 2024-03-29T10:52:34.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f50377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:34.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f50377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:34.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f50377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:34.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f50377fe640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:52:34.110 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f50377fe640 1 -- 172.21.15.5:0/1004071525 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5028003520 con 0x7f50380721c0 2024-03-29T10:52:34.110 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f50377fe640 1 -- 172.21.15.5:0/1004071525 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1915172285 0 0) 0x7f5028003520 con 0x7f50380721c0 2024-03-29T10:52:34.110 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f50377fe640 1 -- 172.21.15.5:0/1004071525 >> v1:172.21.15.177:6789/0 conn(0x7f5038071de0 legacy=0x7f50380aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:34.110 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.127+0000 7f50377fe640 1 -- 172.21.15.5:0/1004071525 >> v1:172.21.15.5:6789/0 conn(0x7f5038145ae0 legacy=0x7f5038145f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 1 monclient: found mon.noname-b 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 1 -- 172.21.15.5:0/1004071525 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50380b0590 con 0x7f50380721c0 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 1 -- 172.21.15.5:0/1004071525 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5024001690 con 0x7f50380721c0 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: dump: 2024-03-29T10:52:34.111 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f503ef6d640 5 monclient: authenticate success, global_id 4439 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f503ef6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 1 -- 172.21.15.5:0/1004071525 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50240017f0 con 0x7f50380721c0 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 1 -- 172.21.15.5:0/1004071525 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5024001c40 con 0x7f50380721c0 2024-03-29T10:52:34.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f50377fe640 10 monclient: dump: 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:34.114 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f503ef6d640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.129+0000 7f503ef6d640 4 set_mon_vals no callback set 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.130+0000 7f503ef6d640 10 monclient: shutdown 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.130+0000 7f503ef6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.130+0000 7f503ef6d640 1 -- 172.21.15.5:0/1004071525 >> v1:172.21.15.5:6790/0 conn(0x7f50380721c0 legacy=0x7f503813f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.130+0000 7f503ef6d640 1 -- 172.21.15.5:0/1004071525 shutdown_connections 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.130+0000 7f503ef6d640 1 -- 172.21.15.5:0/1004071525 >> 172.21.15.5:0/1004071525 conn(0x7f503806b660 msgr2=0x7f503806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.130+0000 7f503ef6d640 1 -- 172.21.15.5:0/1004071525 shutdown_connections 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.130+0000 7f503ef6d640 1 -- 172.21.15.5:0/1004071525 wait complete. 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.131+0000 7f503ef6d640 10 monclient: build_initial_monmap 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.131+0000 7f503ef6d640 10 monclient: monmap: 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:52:34.115 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.132+0000 7f503ef6d640 1 librados: starting msgr at 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.132+0000 7f503ef6d640 1 librados: starting objecter 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.132+0000 7f503ef6d640 1 Processor -- start 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.132+0000 7f503ef6d640 1 -- start start 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.132+0000 7f503ef6d640 1 librados: setting wanted keys 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.132+0000 7f503ef6d640 1 librados: calling monclient init 2024-03-29T10:52:34.116 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.132+0000 7f503ef6d640 10 monclient: init 2024-03-29T10:52:34.117 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.133+0000 7f503ef6d640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:34.117 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.133+0000 7f503ef6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:34.117 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.133+0000 7f503ef6d640 10 monclient(hunting): picked mon.noname-b con 0x7f5038071de0 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.133+0000 7f503ef6d640 10 monclient(hunting): picked mon.noname-c con 0x7f5038145ae0 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.133+0000 7f5037fff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f5038145ae0 0x7f5038131ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:50954/0 (socket says 172.21.15.5:50954) 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f503ef6d640 10 monclient(hunting): picked mon.noname-a con 0x7f5038137eb0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f503ef6d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5038138da0 con 0x7f5038137eb0 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f503ef6d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50381323b0 con 0x7f5038071de0 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f503ef6d640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5038138340 con 0x7f5038145ae0 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f503ef6d640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f503ef6d640 10 monclient(hunting): authenticate will time out at 1167.375456s 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f5037fff640 1 -- 172.21.15.5:0/979445147 learned_addr learned my addr 172.21.15.5:0/979445147 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f50357fa640 10 client.?.objecter ms_handle_connect 0x7f5038145ae0 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f50357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611575753 0 0) 0x7f5038138340 con 0x7f5038145ae0 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f50357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:52:34.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f50357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f50357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f50357fa640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f501c004eb0 con 0x7f5038145ae0 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.134+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 106828568 0 0) 0x7f501c004eb0 con 0x7f5038145ae0 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.135+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5038138340 con 0x7f5038145ae0 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.135+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5024001bc0 con 0x7f5038145ae0 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.135+0000 7f50357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.135+0000 7f50357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.135+0000 7f50357fa640 10 monclient(hunting): dump: 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:34.119 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:34.120 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:34.120 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:34.120 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:34.120 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:34.120 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.135+0000 7f50357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:34.120 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.135+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1922601926 0 0) 0x7f5038138340 con 0x7f5038145ae0 2024-03-29T10:52:34.120 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.135+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 >> v1:172.21.15.177:6789/0 conn(0x7f5038071de0 legacy=0x7f50381371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.120 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.135+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 >> v1:172.21.15.5:6789/0 conn(0x7f5038137eb0 legacy=0x7f50381e2160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:34.123 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 10 monclient: _finish_hunting 0 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 1 monclient: found mon.c 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5038139010 con 0x7f5038145ae0 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 10 monclient: _finish_auth 0 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 10 monclient: _check_auth_tickets 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f503ef6d640 5 monclient: authenticate success, global_id 4442 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f503ef6d640 10 monclient: _renew_subs 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f503ef6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f503ef6d640 1 -- 172.21.15.5:0/979445147 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50381392f0 con 0x7f5038145ae0 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5024004bf0 con 0x7f5038145ae0 2024-03-29T10:52:34.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f5024004f00 con 0x7f5038145ae0 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f50357fa640 10 monclient: dump: 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:34.125 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.137+0000 7f5036ffd640 4 set_mon_vals no callback set 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.139+0000 7f503ef6d640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.139+0000 7f503ef6d640 10 monclient: _renew_subs 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.139+0000 7f503ef6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.139+0000 7f503ef6d640 1 -- 172.21.15.5:0/979445147 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50380acec0 con 0x7f5038145ae0 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.139+0000 7f503ef6d640 1 librados: init done 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.139+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f502401aed0 con 0x7f5038145ae0 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.140+0000 7f503ef6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.140+0000 7f503ef6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.140+0000 7f503ef6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.140+0000 7f503ef6d640 1 -- 172.21.15.5:0/979445147 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5038075f10 con 0x7f5038145ae0 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.140+0000 7f503cd09640 10 monclient: get_auth_request con 0x7f501c06dc50 auth_method 0 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.140+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f5024085f00 con 0x7f5038145ae0 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.141+0000 7f50357fa640 10 client.4442.objecter ms_dispatch 0x7f5038000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:34.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.141+0000 7f50357fa640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:34.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.141+0000 7f50357fa640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:34.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.141+0000 7f50357fa640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-29T10:52:34.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.141+0000 7f50357fa640 10 client.4442.objecter ms_handle_connect 0x7f501c06dc50 2024-03-29T10:52:34.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.148+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50240507b0 con 0x7f5038145ae0 2024-03-29T10:52:34.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.148+0000 7f50357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:34.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.148+0000 7f50357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:34.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.465+0000 7f503ef6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:34.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.465+0000 7f503ef6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:34.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.465+0000 7f503ef6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:34.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.465+0000 7f503ef6d640 1 -- 172.21.15.5:0/979445147 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5038137b70 con 0x7f5038145ae0 2024-03-29T10:52:34.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.466+0000 7f50357fa640 1 -- 172.21.15.5:0/979445147 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f50240526e0 con 0x7f5038145ae0 2024-03-29T10:52:34.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.466+0000 7f50357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:34.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.466+0000 7f50357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:34.448 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:52:34.448 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:34.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.469+0000 7f503ef6d640 10 librados: watch_flush enter 2024-03-29T10:52:34.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.469+0000 7f503ef6d640 10 librados: watch_flush exit 2024-03-29T10:52:34.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.469+0000 7f503ef6d640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-29T10:52:34.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.469+0000 7f503ef6d640 10 client.4442.objecter successfully canceled tick 2024-03-29T10:52:34.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.469+0000 7f503ef6d640 1 -- 172.21.15.5:0/979445147 >> v1:172.21.15.5:6816/2847500151 conn(0x7f501c06dc50 legacy=0x7f501c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.470+0000 7f503ef6d640 10 monclient: shutdown 2024-03-29T10:52:34.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.470+0000 7f503ef6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:34.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.470+0000 7f503ef6d640 1 -- 172.21.15.5:0/979445147 >> v1:172.21.15.5:6790/0 conn(0x7f5038145ae0 legacy=0x7f5038131ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:34.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.470+0000 7f503ef6d640 1 -- 172.21.15.5:0/979445147 shutdown_connections 2024-03-29T10:52:34.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.470+0000 7f503ef6d640 1 -- 172.21.15.5:0/979445147 >> 172.21.15.5:0/979445147 conn(0x7f503806b660 msgr2=0x7f50380a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:34.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.470+0000 7f503ef6d640 1 -- 172.21.15.5:0/979445147 shutdown_connections 2024-03-29T10:52:34.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.470+0000 7f503ef6d640 1 -- 172.21.15.5:0/979445147 wait complete. 2024-03-29T10:52:34.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:34.470+0000 7f503ef6d640 1 librados: shutdown 2024-03-29T10:52:34.468 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:41.276 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:41.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.447+0000 7fd13471e640 10 monclient: get_monmap_and_config 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.447+0000 7fd13471e640 10 monclient: build_initial_monmap 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.447+0000 7fd13471e640 10 monclient: monmap: 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:52:41.448253+0000 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:52:41.448253+0000 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:52:41.429 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:41.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.449+0000 7fd13471e640 1 Processor -- start 2024-03-29T10:52:41.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.449+0000 7fd13471e640 1 -- start start 2024-03-29T10:52:41.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.449+0000 7fd13471e640 10 monclient: init 2024-03-29T10:52:41.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.450+0000 7fd13471e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:41.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.450+0000 7fd13471e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:41.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.450+0000 7fd13471e640 10 monclient(hunting): picked mon.noname-c con 0x7fd12c0b4790 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:41.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.451+0000 7fd13471e640 10 monclient(hunting): picked mon.noname-a con 0x7fd12c0aea90 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:41.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.451+0000 7fd13471e640 10 monclient(hunting): picked mon.noname-b con 0x7fd12c0b39f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:41.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.451+0000 7fd13471e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0b0c60 con 0x7fd12c0aea90 2024-03-29T10:52:41.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.451+0000 7fd13471e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0af020 con 0x7fd12c0b4790 2024-03-29T10:52:41.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.451+0000 7fd13471e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c13fbd0 con 0x7fd12c0b39f0 2024-03-29T10:52:41.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.451+0000 7fd13471e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:41.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.451+0000 7fd13471e640 10 monclient(hunting): authenticate will time out at 1174.692943s 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.453+0000 7fd131cb9640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd12c0aea90 0x7fd12c0af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33067/0 (socket says 172.21.15.5:33067) 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.453+0000 7fd131cb9640 1 -- 172.21.15.5:0/1610861024 learned_addr learned my addr 172.21.15.5:0/1610861024 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.453+0000 7fd1314b8640 1 -- 172.21.15.5:0/1610861024 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406953152 0 0) 0x7fd12c0b0c60 con 0x7fd12c0aea90 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.453+0000 7fd1314b8640 10 monclient(hunting): _negotiate 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.453+0000 7fd1314b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.453+0000 7fd1314b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.453+0000 7fd1314b8640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.453+0000 7fd1314b8640 1 -- 172.21.15.5:0/1610861024 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd118003520 con 0x7fd12c0aea90 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 1 -- 172.21.15.5:0/1610861024 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3985022143 0 0) 0x7fd118003520 con 0x7fd12c0aea90 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 1 -- 172.21.15.5:0/1610861024 >> v1:172.21.15.5:6790/0 conn(0x7fd12c0b39f0 legacy=0x7fd12c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:41.438 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 1 -- 172.21.15.5:0/1610861024 >> v1:172.21.15.177:6789/0 conn(0x7fd12c0b4790 legacy=0x7fd12c0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: _finish_hunting 0 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 1 monclient: found mon.noname-a 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 1 -- 172.21.15.5:0/1610861024 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c13fd60 con 0x7fd12c0aea90 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: _finish_auth 0 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: _check_auth_tickets 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 1 -- 172.21.15.5:0/1610861024 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fd128001690 con 0x7fd12c0aea90 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: dump: 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:41.439 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:41.440 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:41.440 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:41.440 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.440 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:41.440 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:41.440 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:41.440 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:41.440 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:41.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 1 -- 172.21.15.5:0/1610861024 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd128001860 con 0x7fd12c0aea90 2024-03-29T10:52:41.441 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:41.441 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 1 -- 172.21.15.5:0/1610861024 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fd128001cb0 con 0x7fd12c0aea90 2024-03-29T10:52:41.441 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.441 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:41.441 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.454+0000 7fd1314b8640 10 monclient: dump: 2024-03-29T10:52:41.441 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:41.441 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:41.441 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:41.441 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:41.442 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.442 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:41.442 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:41.442 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:41.442 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:41.442 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:41.442 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.455+0000 7fd13471e640 5 monclient: authenticate success, global_id 4386 2024-03-29T10:52:41.442 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.455+0000 7fd13471e640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:41.442 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.455+0000 7fd13471e640 4 set_mon_vals no callback set 2024-03-29T10:52:41.442 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.455+0000 7fd13471e640 10 monclient: shutdown 2024-03-29T10:52:41.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.455+0000 7fd13471e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:41.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.455+0000 7fd13471e640 1 -- 172.21.15.5:0/1610861024 >> v1:172.21.15.5:6789/0 conn(0x7fd12c0aea90 legacy=0x7fd12c0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.456+0000 7fd13471e640 1 -- 172.21.15.5:0/1610861024 shutdown_connections 2024-03-29T10:52:41.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.456+0000 7fd13471e640 1 -- 172.21.15.5:0/1610861024 >> 172.21.15.5:0/1610861024 conn(0x7fd12c066a00 msgr2=0x7fd12c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:41.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.458+0000 7fd13471e640 1 -- 172.21.15.5:0/1610861024 shutdown_connections 2024-03-29T10:52:41.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.458+0000 7fd13471e640 1 -- 172.21.15.5:0/1610861024 wait complete. 2024-03-29T10:52:41.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.459+0000 7fd13471e640 10 monclient: build_initial_monmap 2024-03-29T10:52:41.446 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.459+0000 7fd13471e640 10 monclient: monmap: 2024-03-29T10:52:41.446 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.459+0000 7fd13471e640 1 librados: starting msgr at 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.459+0000 7fd13471e640 1 librados: starting objecter 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.460+0000 7fd13471e640 1 Processor -- start 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.460+0000 7fd13471e640 1 -- start start 2024-03-29T10:52:41.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.460+0000 7fd13471e640 1 librados: setting wanted keys 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.460+0000 7fd13471e640 1 librados: calling monclient init 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.460+0000 7fd13471e640 10 monclient: init 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.461+0000 7fd13471e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.461+0000 7fd13471e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.461+0000 7fd13471e640 10 monclient(hunting): picked mon.noname-a con 0x7fd12c0b39f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.462+0000 7fd13471e640 10 monclient(hunting): picked mon.noname-b con 0x7fd12c0b4790 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.462+0000 7fd13471e640 10 monclient(hunting): picked mon.noname-c con 0x7fd12c142470 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.462+0000 7fd13471e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c131240 con 0x7fd12c0b39f0 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.462+0000 7fd13471e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c1318b0 con 0x7fd12c0b4790 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.462+0000 7fd13471e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c142f30 con 0x7fd12c142470 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.462+0000 7fd13471e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.462+0000 7fd13471e640 10 monclient(hunting): authenticate will time out at 1174.703287s 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.462+0000 7fd132cbb640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fd12c142470 0x7fd12c1e1ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:47972/0 (socket says 172.21.15.5:47972) 2024-03-29T10:52:41.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.462+0000 7fd132cbb640 1 -- 172.21.15.5:0/1159062449 learned_addr learned my addr 172.21.15.5:0/1159062449 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.463+0000 7fd122ffd640 10 client.?.objecter ms_handle_connect 0x7fd12c142470 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.463+0000 7fd122ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716713090 0 0) 0x7fd12c142f30 con 0x7fd12c142470 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd110004eb0 con 0x7fd12c142470 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3773970346 0 0) 0x7fd110004eb0 con 0x7fd12c142470 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd12c142f30 con 0x7fd12c142470 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fd11c0018e0 con 0x7fd12c142470 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:41.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.464+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3992419296 0 0) 0x7fd12c142f30 con 0x7fd12c142470 2024-03-29T10:52:41.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 >> v1:172.21.15.177:6789/0 conn(0x7fd12c0b4790 legacy=0x7fd12c140350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:41.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 >> v1:172.21.15.5:6789/0 conn(0x7fd12c0b39f0 legacy=0x7fd12c12ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:41.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 1 monclient: found mon.c 2024-03-29T10:52:41.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:41.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:41.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c135530 con 0x7fd12c142470 2024-03-29T10:52:41.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 10 client.?.objecter ms_handle_connect 0x7fd12c0b39f0 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.465+0000 7fd122ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd11c0035f0 con 0x7fd12c142470 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd122ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fd11c003a00 con 0x7fd12c142470 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd122ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd122ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd122ffd640 10 monclient: dump: 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:41.454 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd13471e640 5 monclient: authenticate success, global_id 4445 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd13471e640 10 monclient: _renew_subs 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd13471e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd13471e640 1 -- 172.21.15.5:0/1159062449 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c140a60 con 0x7fd12c142470 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd13471e640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.466+0000 7fd13471e640 10 monclient: _renew_subs 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.467+0000 7fd13471e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:41.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.467+0000 7fd13471e640 1 -- 172.21.15.5:0/1159062449 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c140f90 con 0x7fd12c142470 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.467+0000 7fd13471e640 1 librados: init done 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.468+0000 7fd123fff640 4 set_mon_vals no callback set 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.469+0000 7fd13471e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.469+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fd11c003be0 con 0x7fd12c142470 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.470+0000 7fd13471e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.470+0000 7fd13471e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.470+0000 7fd13471e640 1 -- 172.21.15.5:0/1159062449 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd114005080 con 0x7fd12c142470 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.470+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fd11c084a70 con 0x7fd12c142470 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.470+0000 7fd122ffd640 10 client.4445.objecter ms_dispatch 0x7fd12c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.470+0000 7fd122ffd640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.470+0000 7fd122ffd640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.470+0000 7fd122ffd640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.470+0000 7fd1324ba640 10 monclient: get_auth_request con 0x7fd11006db10 auth_method 0 2024-03-29T10:52:41.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.470+0000 7fd122ffd640 10 client.4445.objecter ms_handle_connect 0x7fd11006db10 2024-03-29T10:52:41.461 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.480+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd11c0503b0 con 0x7fd12c142470 2024-03-29T10:52:41.461 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.480+0000 7fd122ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:41.461 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.480+0000 7fd122ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:41.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.813+0000 7fd13471e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:41.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.813+0000 7fd13471e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:41.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.813+0000 7fd13471e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:41.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.813+0000 7fd13471e640 1 -- 172.21.15.5:0/1159062449 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd114005330 con 0x7fd12c142470 2024-03-29T10:52:41.798 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:52:41.798 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:41.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.816+0000 7fd122ffd640 1 -- 172.21.15.5:0/1159062449 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fd11c0522e0 con 0x7fd12c142470 2024-03-29T10:52:41.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.816+0000 7fd122ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:41.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.816+0000 7fd122ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:41.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.821+0000 7fd13471e640 10 librados: watch_flush enter 2024-03-29T10:52:41.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.821+0000 7fd13471e640 10 librados: watch_flush exit 2024-03-29T10:52:41.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.821+0000 7fd13471e640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T10:52:41.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.821+0000 7fd13471e640 10 client.4445.objecter successfully canceled tick 2024-03-29T10:52:41.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.821+0000 7fd13471e640 1 -- 172.21.15.5:0/1159062449 >> v1:172.21.15.5:6816/2847500151 conn(0x7fd11006db10 legacy=0x7fd11006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.821+0000 7fd13471e640 10 monclient: shutdown 2024-03-29T10:52:41.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.821+0000 7fd13471e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:41.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.821+0000 7fd13471e640 1 -- 172.21.15.5:0/1159062449 >> v1:172.21.15.5:6790/0 conn(0x7fd12c142470 legacy=0x7fd12c1e1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:41.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.822+0000 7fd13471e640 1 -- 172.21.15.5:0/1159062449 shutdown_connections 2024-03-29T10:52:41.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.822+0000 7fd13471e640 1 -- 172.21.15.5:0/1159062449 >> 172.21.15.5:0/1159062449 conn(0x7fd12c066a00 msgr2=0x7fd12c0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:41.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.823+0000 7fd13471e640 1 -- 172.21.15.5:0/1159062449 shutdown_connections 2024-03-29T10:52:41.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.825+0000 7fd13471e640 1 -- 172.21.15.5:0/1159062449 wait complete. 2024-03-29T10:52:41.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:41.825+0000 7fd13471e640 1 librados: shutdown 2024-03-29T10:52:41.823 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:48.632 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:48.781 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.799+0000 7f7f544f8640 10 monclient: get_monmap_and_config 2024-03-29T10:52:48.781 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.799+0000 7f7f544f8640 10 monclient: build_initial_monmap 2024-03-29T10:52:48.781 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.800+0000 7f7f544f8640 10 monclient: monmap: 2024-03-29T10:52:48.781 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:48.781 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:48.782 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:52:48.801019+0000 2024-03-29T10:52:48.782 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:52:48.801019+0000 2024-03-29T10:52:48.782 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:48.782 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:48.782 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:48.782 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:52:48.782 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:52:48.782 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:48.783 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.800+0000 7f7f544f8640 1 Processor -- start 2024-03-29T10:52:48.783 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.801+0000 7f7f544f8640 1 -- start start 2024-03-29T10:52:48.783 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.801+0000 7f7f544f8640 10 monclient: init 2024-03-29T10:52:48.786 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.802+0000 7f7f544f8640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:48.786 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.802+0000 7f7f544f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:48.786 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.802+0000 7f7f544f8640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4c071de0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:48.786 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.805+0000 7f7f544f8640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4c0721c0 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:48.787 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.805+0000 7f7f544f8640 10 monclient(hunting): picked mon.noname-b con 0x7f7f4c1425d0 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:48.787 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.805+0000 7f7f52294640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7f4c071de0 0x7f7f4c13f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:40475/0 (socket says 172.21.15.5:40475) 2024-03-29T10:52:48.787 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.805+0000 7f7f52294640 1 -- 172.21.15.5:0/2820517187 learned_addr learned my addr 172.21.15.5:0/2820517187 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:48.787 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.805+0000 7f7f544f8640 1 -- 172.21.15.5:0/2820517187 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c13fb10 con 0x7f7f4c071de0 2024-03-29T10:52:48.787 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.805+0000 7f7f544f8640 1 -- 172.21.15.5:0/2820517187 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c142c20 con 0x7f7f4c0721c0 2024-03-29T10:52:48.787 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.805+0000 7f7f544f8640 1 -- 172.21.15.5:0/2820517187 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c0ada50 con 0x7f7f4c1425d0 2024-03-29T10:52:48.787 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.805+0000 7f7f544f8640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:48.787 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.805+0000 7f7f544f8640 10 monclient(hunting): authenticate will time out at 1182.046910s 2024-03-29T10:52:48.788 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913907246 0 0) 0x7f7f4c0ada50 con 0x7f7f4c1425d0 2024-03-29T10:52:48.788 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 10 monclient(hunting): _negotiate 2024-03-29T10:52:48.788 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:48.788 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:48.789 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:52:48.789 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f38003520 con 0x7f7f4c1425d0 2024-03-29T10:52:48.789 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789905413 0 0) 0x7f7f4c13fb10 con 0x7f7f4c071de0 2024-03-29T10:52:48.789 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 10 monclient(hunting): _negotiate 2024-03-29T10:52:48.789 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:48.789 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:48.789 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:52:48.789 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f4c0ada50 con 0x7f7f4c071de0 2024-03-29T10:52:48.789 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 62061432 0 0) 0x7f7f38003520 con 0x7f7f4c1425d0 2024-03-29T10:52:48.789 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.806+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 >> v1:172.21.15.177:6789/0 conn(0x7f7f4c0721c0 legacy=0x7f7f4c145670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:48.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.809+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 >> v1:172.21.15.5:6789/0 conn(0x7f7f4c071de0 legacy=0x7f7f4c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.810+0000 7f7f51292640 10 monclient: _finish_hunting 0 2024-03-29T10:52:48.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.810+0000 7f7f51292640 1 monclient: found mon.noname-b 2024-03-29T10:52:48.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.810+0000 7f7f51292640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:48.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.810+0000 7f7f51292640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:52:48.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.810+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c0b0500 con 0x7f7f4c1425d0 2024-03-29T10:52:48.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.810+0000 7f7f51292640 10 monclient: _finish_auth 0 2024-03-29T10:52:48.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.810+0000 7f7f51292640 10 monclient: _check_auth_tickets 2024-03-29T10:52:48.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:48.794 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f544f8640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f544f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7f44001940 con 0x7f7f4c1425d0 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 10 monclient: dump: 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:48.795 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f544f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f44001b60 con 0x7f7f4c1425d0 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f544f8640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f544f8640 4 set_mon_vals no callback set 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 1 -- 172.21.15.5:0/2820517187 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7f44001ef0 con 0x7f7f4c1425d0 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.811+0000 7f7f51292640 10 monclient: dump: 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.796 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:48.797 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:48.797 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:48.797 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:48.797 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:48.797 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.812+0000 7f7f544f8640 10 monclient: shutdown 2024-03-29T10:52:48.797 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.812+0000 7f7f544f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:48.797 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.812+0000 7f7f544f8640 1 -- 172.21.15.5:0/2820517187 >> v1:172.21.15.5:6790/0 conn(0x7f7f4c1425d0 legacy=0x7f7f4c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.797 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.812+0000 7f7f544f8640 1 -- 172.21.15.5:0/2820517187 shutdown_connections 2024-03-29T10:52:48.797 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.812+0000 7f7f544f8640 1 -- 172.21.15.5:0/2820517187 >> 172.21.15.5:0/2820517187 conn(0x7f7f4c06b660 msgr2=0x7f7f4c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:48.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.815+0000 7f7f544f8640 1 -- 172.21.15.5:0/2820517187 shutdown_connections 2024-03-29T10:52:48.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.815+0000 7f7f544f8640 1 -- 172.21.15.5:0/2820517187 wait complete. 2024-03-29T10:52:48.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.816+0000 7f7f544f8640 10 monclient: build_initial_monmap 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.818+0000 7f7f544f8640 10 monclient: monmap: 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:52:48.800 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:48.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.818+0000 7f7f544f8640 1 librados: starting msgr at 2024-03-29T10:52:48.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.818+0000 7f7f544f8640 1 librados: starting objecter 2024-03-29T10:52:48.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.818+0000 7f7f544f8640 1 Processor -- start 2024-03-29T10:52:48.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.818+0000 7f7f544f8640 1 -- start start 2024-03-29T10:52:48.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.818+0000 7f7f544f8640 1 librados: setting wanted keys 2024-03-29T10:52:48.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.819+0000 7f7f544f8640 1 librados: calling monclient init 2024-03-29T10:52:48.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.819+0000 7f7f544f8640 10 monclient: init 2024-03-29T10:52:48.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.819+0000 7f7f544f8640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:48.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.819+0000 7f7f544f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:48.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f544f8640 10 monclient(hunting): picked mon.noname-b con 0x7f7f4c071de0 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:48.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f52294640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f7f4c071de0 0x7f7f4c13b4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:37823/0 (socket says 172.21.15.5:37823) 2024-03-29T10:52:48.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f544f8640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4c1425d0 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:48.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f544f8640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4c135fc0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:48.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f544f8640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c13cc20 con 0x7f7f4c135fc0 2024-03-29T10:52:48.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f544f8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c13cf00 con 0x7f7f4c071de0 2024-03-29T10:52:48.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f544f8640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c13af90 con 0x7f7f4c1425d0 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f544f8640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f544f8640 10 monclient(hunting): authenticate will time out at 1182.063589s 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f51a93640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f7f4c1425d0 0x7f7f4c1358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:51360/0 (socket says 172.21.15.5:51360) 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f51a93640 1 -- 172.21.15.5:0/573929808 learned_addr learned my addr 172.21.15.5:0/573929808 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f42ffd640 10 client.?.objecter ms_handle_connect 0x7f7f4c1425d0 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584415365 0 0) 0x7f7f4c13af90 con 0x7f7f4c1425d0 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f42ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.822+0000 7f7f42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f30004eb0 con 0x7f7f4c1425d0 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 10 client.?.objecter ms_handle_connect 0x7f7f4c071de0 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:48.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2902374084 0 0) 0x7f7f30004eb0 con 0x7f7f4c1425d0 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f4c13af90 con 0x7f7f4c1425d0 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7f3c001870 con 0x7f7f4c1425d0 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:48.808 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836448859 0 0) 0x7f7f4c13cf00 con 0x7f7f4c071de0 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.823+0000 7f7f42ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.824+0000 7f7f42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.824+0000 7f7f42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.824+0000 7f7f42ffd640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.824+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f30004eb0 con 0x7f7f4c071de0 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.824+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2242156471 0 0) 0x7f7f4c13af90 con 0x7f7f4c1425d0 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.824+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 >> v1:172.21.15.177:6789/0 conn(0x7f7f4c071de0 legacy=0x7f7f4c13b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.809 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.824+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 >> v1:172.21.15.5:6789/0 conn(0x7f7f4c135fc0 legacy=0x7f7f4c1e2700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:48.813 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f42ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:48.813 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f42ffd640 1 monclient: found mon.c 2024-03-29T10:52:48.813 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:48.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f42ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:48.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c13f930 con 0x7f7f4c1425d0 2024-03-29T10:52:48.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f42ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:48.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f42ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:48.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:48.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f544f8640 5 monclient: authenticate success, global_id 4451 2024-03-29T10:52:48.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f544f8640 10 monclient: _renew_subs 2024-03-29T10:52:48.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f544f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:48.814 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f544f8640 1 -- 172.21.15.5:0/573929808 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c13bcf0 con 0x7f7f4c1425d0 2024-03-29T10:52:48.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f544f8640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:48.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f544f8640 10 monclient: _renew_subs 2024-03-29T10:52:48.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f544f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:48.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f544f8640 1 -- 172.21.15.5:0/573929808 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c141a20 con 0x7f7f4c1425d0 2024-03-29T10:52:48.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.826+0000 7f7f544f8640 1 librados: init done 2024-03-29T10:52:48.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.828+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f3c0019d0 con 0x7f7f4c1425d0 2024-03-29T10:52:48.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.828+0000 7f7f42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:48.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.828+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7f3c004aa0 con 0x7f7f4c1425d0 2024-03-29T10:52:48.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.828+0000 7f7f42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:48.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.828+0000 7f7f42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:48.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.828+0000 7f7f42ffd640 10 monclient: dump: 2024-03-29T10:52:48.816 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:48.816 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:48.816 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:48.816 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:48.816 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:48.816 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:48.818 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:48.818 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:48.818 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:48.818 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:48.818 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.828+0000 7f7f50a91640 4 set_mon_vals no callback set 2024-03-29T10:52:48.818 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.828+0000 7f7f544f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:48.818 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.830+0000 7f7f544f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:48.818 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.830+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f7f3c004c00 con 0x7f7f4c1425d0 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.831+0000 7f7f544f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.831+0000 7f7f544f8640 1 -- 172.21.15.5:0/573929808 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c075f10 con 0x7f7f4c1425d0 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.831+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f7f3c084af0 con 0x7f7f4c1425d0 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.831+0000 7f7f42ffd640 10 client.4451.objecter ms_dispatch 0x7f7f4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.831+0000 7f7f42ffd640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:48.819 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.831+0000 7f7f42ffd640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:48.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.831+0000 7f7f42ffd640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T10:52:48.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.832+0000 7f7f52294640 10 monclient: get_auth_request con 0x7f7f3006dd30 auth_method 0 2024-03-29T10:52:48.820 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.832+0000 7f7f42ffd640 10 client.4451.objecter ms_handle_connect 0x7f7f3006dd30 2024-03-29T10:52:48.821 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.839+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f3c0503d0 con 0x7f7f4c1425d0 2024-03-29T10:52:48.821 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.839+0000 7f7f42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:48.821 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:48.839+0000 7f7f42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:49.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.171+0000 7f7f544f8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:49.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.171+0000 7f7f544f8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:49.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.171+0000 7f7f544f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:52:49.153 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.171+0000 7f7f544f8640 1 -- 172.21.15.5:0/573929808 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f4c13fac0 con 0x7f7f4c1425d0 2024-03-29T10:52:49.156 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:52:49.157 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:49.157 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.174+0000 7f7f42ffd640 1 -- 172.21.15.5:0/573929808 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f7f3c052300 con 0x7f7f4c1425d0 2024-03-29T10:52:49.157 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.174+0000 7f7f42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:49.157 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.174+0000 7f7f42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:49.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.178+0000 7f7f544f8640 10 librados: watch_flush enter 2024-03-29T10:52:49.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.180+0000 7f7f544f8640 10 librados: watch_flush exit 2024-03-29T10:52:49.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.180+0000 7f7f544f8640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T10:52:49.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.180+0000 7f7f544f8640 10 client.4451.objecter successfully canceled tick 2024-03-29T10:52:49.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.180+0000 7f7f544f8640 1 -- 172.21.15.5:0/573929808 >> v1:172.21.15.5:6816/2847500151 conn(0x7f7f3006dd30 legacy=0x7f7f300701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:49.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.180+0000 7f7f544f8640 10 monclient: shutdown 2024-03-29T10:52:49.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.181+0000 7f7f544f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:49.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.181+0000 7f7f544f8640 1 -- 172.21.15.5:0/573929808 >> v1:172.21.15.5:6790/0 conn(0x7f7f4c1425d0 legacy=0x7f7f4c1358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:49.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.181+0000 7f7f544f8640 1 -- 172.21.15.5:0/573929808 shutdown_connections 2024-03-29T10:52:49.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.181+0000 7f7f544f8640 1 -- 172.21.15.5:0/573929808 >> 172.21.15.5:0/573929808 conn(0x7f7f4c06b660 msgr2=0x7f7f4c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:49.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.181+0000 7f7f544f8640 1 -- 172.21.15.5:0/573929808 shutdown_connections 2024-03-29T10:52:49.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.181+0000 7f7f544f8640 1 -- 172.21.15.5:0/573929808 wait complete. 2024-03-29T10:52:49.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:49.181+0000 7f7f544f8640 1 librados: shutdown 2024-03-29T10:52:49.183 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:55.991 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:56.136 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.154+0000 7fbf7884e640 10 monclient: get_monmap_and_config 2024-03-29T10:52:56.136 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.154+0000 7fbf7884e640 10 monclient: build_initial_monmap 2024-03-29T10:52:56.136 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.155+0000 7fbf7884e640 10 monclient: monmap: 2024-03-29T10:52:56.136 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:56.136 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:56.136 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:52:56.155985+0000 2024-03-29T10:52:56.136 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:52:56.155985+0000 2024-03-29T10:52:56.136 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:56.137 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:56.137 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:56.137 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:52:56.137 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:52:56.137 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:56.137 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.155+0000 7fbf7884e640 1 Processor -- start 2024-03-29T10:52:56.138 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.155+0000 7fbf7884e640 1 -- start start 2024-03-29T10:52:56.139 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.155+0000 7fbf7884e640 10 monclient: init 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.156+0000 7fbf7884e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.156+0000 7fbf7884e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.156+0000 7fbf7884e640 10 monclient(hunting): picked mon.noname-c con 0x7fbf741457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.156+0000 7fbf7884e640 10 monclient(hunting): picked mon.noname-b con 0x7fbf7413f650 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.156+0000 7fbf7884e640 10 monclient(hunting): picked mon.noname-a con 0x7fbf740a9d70 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.157+0000 7fbf7884e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf740aaef0 con 0x7fbf740a9d70 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.157+0000 7fbf7884e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf74145bd0 con 0x7fbf741457f0 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.157+0000 7fbf7884e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf74145eb0 con 0x7fbf7413f650 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.157+0000 7fbf7884e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.157+0000 7fbf7884e640 10 monclient(hunting): authenticate will time out at 1189.398114s 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.157+0000 7fbf71d9b640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fbf7413f650 0x7fbf740b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:45714/0 (socket says 172.21.15.5:45714) 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.157+0000 7fbf71d9b640 1 -- 172.21.15.5:0/94244302 learned_addr learned my addr 172.21.15.5:0/94244302 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:56.140 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.158+0000 7fbf7159a640 1 -- 172.21.15.5:0/94244302 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319371389 0 0) 0x7fbf74145eb0 con 0x7fbf7413f650 2024-03-29T10:52:56.141 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.158+0000 7fbf7159a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:56.141 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.158+0000 7fbf7159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:56.141 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.158+0000 7fbf7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:56.141 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.158+0000 7fbf7159a640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:52:56.141 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.158+0000 7fbf7159a640 1 -- 172.21.15.5:0/94244302 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c003520 con 0x7fbf7413f650 2024-03-29T10:52:56.141 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.158+0000 7fbf7159a640 1 -- 172.21.15.5:0/94244302 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 263691160 0 0) 0x7fbf5c003520 con 0x7fbf7413f650 2024-03-29T10:52:56.141 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.158+0000 7fbf7159a640 1 -- 172.21.15.5:0/94244302 >> v1:172.21.15.177:6789/0 conn(0x7fbf741457f0 legacy=0x7fbf7413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:56.145 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 1 -- 172.21.15.5:0/94244302 >> v1:172.21.15.5:6789/0 conn(0x7fbf740a9d70 legacy=0x7fbf7414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:56.145 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 10 monclient: _finish_hunting 0 2024-03-29T10:52:56.145 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 1 monclient: found mon.noname-b 2024-03-29T10:52:56.145 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:56.145 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:52:56.145 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 1 -- 172.21.15.5:0/94244302 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf74146340 con 0x7fbf7413f650 2024-03-29T10:52:56.145 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 10 monclient: _finish_auth 0 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 10 monclient: _check_auth_tickets 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7884e640 5 monclient: authenticate success, global_id 4454 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7884e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 1 -- 172.21.15.5:0/94244302 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbf68001690 con 0x7fbf7413f650 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7159a640 10 monclient: dump: 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:56.146 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.160+0000 7fbf7884e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.161+0000 7fbf7159a640 1 -- 172.21.15.5:0/94244302 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf680019a0 con 0x7fbf7413f650 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.161+0000 7fbf7159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.161+0000 7fbf7884e640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.161+0000 7fbf7884e640 4 set_mon_vals no callback set 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7159a640 1 -- 172.21.15.5:0/94244302 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbf68001d30 con 0x7fbf7413f650 2024-03-29T10:52:56.147 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7159a640 10 monclient: dump: 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7884e640 10 monclient: shutdown 2024-03-29T10:52:56.148 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7884e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:56.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7884e640 1 -- 172.21.15.5:0/94244302 >> v1:172.21.15.5:6790/0 conn(0x7fbf7413f650 legacy=0x7fbf740b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7884e640 1 -- 172.21.15.5:0/94244302 shutdown_connections 2024-03-29T10:52:56.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7884e640 1 -- 172.21.15.5:0/94244302 >> 172.21.15.5:0/94244302 conn(0x7fbf74066a00 msgr2=0x7fbf74066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:56.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7884e640 1 -- 172.21.15.5:0/94244302 shutdown_connections 2024-03-29T10:52:56.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.162+0000 7fbf7884e640 1 -- 172.21.15.5:0/94244302 wait complete. 2024-03-29T10:52:56.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.163+0000 7fbf7884e640 10 monclient: build_initial_monmap 2024-03-29T10:52:56.149 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.163+0000 7fbf7884e640 10 monclient: monmap: 2024-03-29T10:52:56.149 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:52:56.149 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:56.149 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:52:56.150 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:52:56.150 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:56.150 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:56.150 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:52:56.150 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:52:56.150 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:52:56.150 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:56.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.164+0000 7fbf7884e640 1 librados: starting msgr at 2024-03-29T10:52:56.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.164+0000 7fbf7884e640 1 librados: starting objecter 2024-03-29T10:52:56.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.164+0000 7fbf7884e640 1 Processor -- start 2024-03-29T10:52:56.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.165+0000 7fbf7884e640 1 -- start start 2024-03-29T10:52:56.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.165+0000 7fbf7884e640 1 librados: setting wanted keys 2024-03-29T10:52:56.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.165+0000 7fbf7884e640 1 librados: calling monclient init 2024-03-29T10:52:56.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.165+0000 7fbf7884e640 10 monclient: init 2024-03-29T10:52:56.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7884e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:56.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7884e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:56.158 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7884e640 10 monclient(hunting): picked mon.noname-c con 0x7fbf740a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7884e640 10 monclient(hunting): picked mon.noname-b con 0x7fbf741457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7884e640 10 monclient(hunting): picked mon.noname-a con 0x7fbf7413fda0 addr v1:172.21.15.5:6789/0 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7884e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7406fd50 con 0x7fbf7413fda0 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7884e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf741e3db0 con 0x7fbf741457f0 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7884e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf741e49e0 con 0x7fbf740a9d70 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7884e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7884e640 10 monclient(hunting): authenticate will time out at 1189.407533s 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7259c640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fbf740a9d70 0x7fbf74142890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:50504/0 (socket says 172.21.15.5:50504) 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.166+0000 7fbf7259c640 1 -- 172.21.15.5:0/1335096586 learned_addr learned my addr 172.21.15.5:0/1335096586 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.167+0000 7fbf62ffd640 10 client.?.objecter ms_handle_connect 0x7fbf740a9d70 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.167+0000 7fbf62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.169+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143716064 0 0) 0x7fbf741e49e0 con 0x7fbf740a9d70 2024-03-29T10:52:56.159 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.170+0000 7fbf62ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.170+0000 7fbf62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.170+0000 7fbf62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.170+0000 7fbf62ffd640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.170+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf54004eb0 con 0x7fbf740a9d70 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.170+0000 7fbf62ffd640 10 client.?.objecter ms_handle_connect 0x7fbf7413fda0 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.170+0000 7fbf62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718027899 0 0) 0x7fbf7406fd50 con 0x7fbf7413fda0 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf741e49e0 con 0x7fbf7413fda0 2024-03-29T10:52:56.160 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 239474414 0 0) 0x7fbf741e49e0 con 0x7fbf7413fda0 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf7406fd50 con 0x7fbf7413fda0 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbf6c001b30 con 0x7fbf7413fda0 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:56.161 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.171+0000 7fbf62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1370815913 0 0) 0x7fbf54004eb0 con 0x7fbf740a9d70 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf741e49e0 con 0x7fbf740a9d70 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbf640017d0 con 0x7fbf740a9d70 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:56.162 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3659014043 0 0) 0x7fbf7406fd50 con 0x7fbf7413fda0 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 >> v1:172.21.15.5:6790/0 conn(0x7fbf740a9d70 legacy=0x7fbf74142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 >> v1:172.21.15.177:6789/0 conn(0x7fbf741457f0 legacy=0x7fbf741e42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 1 monclient: found mon.a 2024-03-29T10:52:56.163 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf741e4b70 con 0x7fbf7413fda0 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf7884e640 5 monclient: authenticate success, global_id 4395 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf7884e640 10 monclient: _renew_subs 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf7884e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf7884e640 1 -- 172.21.15.5:0/1335096586 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7406e100 con 0x7fbf7413fda0 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf7884e640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf7884e640 10 monclient: _renew_subs 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf7884e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf7884e640 1 -- 172.21.15.5:0/1335096586 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf741e14b0 con 0x7fbf7413fda0 2024-03-29T10:52:56.164 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.172+0000 7fbf7884e640 1 librados: init done 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.173+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf6c002040 con 0x7fbf7413fda0 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.173+0000 7fbf62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.173+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbf6c003b10 con 0x7fbf7413fda0 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.173+0000 7fbf62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.173+0000 7fbf62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.173+0000 7fbf62ffd640 10 monclient: dump: 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:52:56.165 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:52:56.166 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:52:56.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.174+0000 7fbf70d99640 4 set_mon_vals no callback set 2024-03-29T10:52:56.166 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.174+0000 7fbf7884e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:56.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.175+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fbf6c019ae0 con 0x7fbf7413fda0 2024-03-29T10:52:56.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.175+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fbf6c083a90 con 0x7fbf7413fda0 2024-03-29T10:52:56.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.175+0000 7fbf62ffd640 10 client.4395.objecter ms_dispatch 0x7fbf74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:56.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.175+0000 7fbf62ffd640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:56.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.175+0000 7fbf62ffd640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:56.167 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.175+0000 7fbf62ffd640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-29T10:52:56.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.176+0000 7fbf7884e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:56.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.176+0000 7fbf7884e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:52:56.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.176+0000 7fbf7884e640 1 -- 172.21.15.5:0/1335096586 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7406fbc0 con 0x7fbf7413fda0 2024-03-29T10:52:56.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.176+0000 7fbf7259c640 10 monclient: get_auth_request con 0x7fbf5406dee0 auth_method 0 2024-03-29T10:52:56.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.176+0000 7fbf62ffd640 10 client.4395.objecter ms_handle_connect 0x7fbf5406dee0 2024-03-29T10:52:56.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.184+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf6c04f1a0 con 0x7fbf7413fda0 2024-03-29T10:52:56.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.184+0000 7fbf62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:56.168 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.184+0000 7fbf62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:56.489 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.508+0000 7fbf7884e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:56.489 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.508+0000 7fbf7884e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:56.489 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.508+0000 7fbf7884e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:52:56.490 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.508+0000 7fbf7884e640 1 -- 172.21.15.5:0/1335096586 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf74075f10 con 0x7fbf7413fda0 2024-03-29T10:52:56.490 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.509+0000 7fbf62ffd640 1 -- 172.21.15.5:0/1335096586 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fbf6c0512a0 con 0x7fbf7413fda0 2024-03-29T10:52:56.490 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.509+0000 7fbf62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:56.490 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.509+0000 7fbf62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:56.491 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:52:56.491 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:56.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.512+0000 7fbf7884e640 10 librados: watch_flush enter 2024-03-29T10:52:56.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.512+0000 7fbf7884e640 10 librados: watch_flush exit 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.512+0000 7fbf7884e640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.512+0000 7fbf7884e640 10 client.4395.objecter successfully canceled tick 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.512+0000 7fbf7884e640 1 -- 172.21.15.5:0/1335096586 >> v1:172.21.15.5:6816/2847500151 conn(0x7fbf5406dee0 legacy=0x7fbf540703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.512+0000 7fbf7884e640 10 monclient: shutdown 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.512+0000 7fbf7884e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.512+0000 7fbf7884e640 1 -- 172.21.15.5:0/1335096586 >> v1:172.21.15.5:6789/0 conn(0x7fbf7413fda0 legacy=0x7fbf741e0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.513+0000 7fbf7884e640 1 -- 172.21.15.5:0/1335096586 shutdown_connections 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.513+0000 7fbf7884e640 1 -- 172.21.15.5:0/1335096586 >> 172.21.15.5:0/1335096586 conn(0x7fbf74066a00 msgr2=0x7fbf74066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.513+0000 7fbf7884e640 1 -- 172.21.15.5:0/1335096586 shutdown_connections 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.513+0000 7fbf7884e640 1 -- 172.21.15.5:0/1335096586 wait complete. 2024-03-29T10:52:56.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:52:56.513+0000 7fbf7884e640 1 librados: shutdown 2024-03-29T10:52:56.510 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:03.318 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:03.463 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.482+0000 7f7cda616640 10 monclient: get_monmap_and_config 2024-03-29T10:53:03.463 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.482+0000 7f7cda616640 10 monclient: build_initial_monmap 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.482+0000 7f7cda616640 10 monclient: monmap: 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:53:03.483547+0000 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:53:03.483547+0000 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:53:03.464 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:03.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.484+0000 7f7cda616640 1 Processor -- start 2024-03-29T10:53:03.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.484+0000 7f7cda616640 1 -- start start 2024-03-29T10:53:03.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.484+0000 7f7cda616640 10 monclient: init 2024-03-29T10:53:03.466 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.484+0000 7f7cda616640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:03.466 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.485+0000 7f7cda616640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:03.466 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.485+0000 7f7cda616640 10 monclient(hunting): picked mon.noname-c con 0x7f7cd41422b0 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:03.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.488+0000 7f7cd37fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7cd4142690 0x7f7cd40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50109/0 (socket says 172.21.15.5:50109) 2024-03-29T10:53:03.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.488+0000 7f7cd37fe640 1 -- 172.21.15.5:0/226866574 learned_addr learned my addr 172.21.15.5:0/226866574 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:03.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.488+0000 7f7cda616640 10 monclient(hunting): picked mon.noname-a con 0x7f7cd4142690 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:03.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.488+0000 7f7cda616640 10 monclient(hunting): picked mon.noname-b con 0x7f7cd40b17d0 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:03.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.489+0000 7f7cda616640 1 -- 172.21.15.5:0/226866574 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd4142c80 con 0x7f7cd4142690 2024-03-29T10:53:03.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.489+0000 7f7cda616640 1 -- 172.21.15.5:0/226866574 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd40aae40 con 0x7f7cd41422b0 2024-03-29T10:53:03.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.489+0000 7f7cda616640 1 -- 172.21.15.5:0/226866574 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd413fbd0 con 0x7f7cd40b17d0 2024-03-29T10:53:03.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.489+0000 7f7cda616640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:03.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.489+0000 7f7cda616640 10 monclient(hunting): authenticate will time out at 1196.730443s 2024-03-29T10:53:03.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415098705 0 0) 0x7f7cd4142c80 con 0x7f7cd4142690 2024-03-29T10:53:03.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc003520 con 0x7f7cd4142690 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243024185 0 0) 0x7f7cd413fbd0 con 0x7f7cd40b17d0 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd4142c80 con 0x7f7cd40b17d0 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195353186 0 0) 0x7f7cd40aae40 con 0x7f7cd41422b0 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.491+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd413fbd0 con 0x7f7cd41422b0 2024-03-29T10:53:03.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1351173280 0 0) 0x7f7cd413fbd0 con 0x7f7cd41422b0 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 >> v1:172.21.15.5:6790/0 conn(0x7f7cd40b17d0 legacy=0x7f7cd40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 >> v1:172.21.15.5:6789/0 conn(0x7f7cd4142690 legacy=0x7f7cd40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 1 monclient: found mon.noname-c 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd4150930 con 0x7f7cd41422b0 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cda616640 5 monclient: authenticate success, global_id 4372 2024-03-29T10:53:03.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cda616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:03.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7cc4001460 con 0x7f7cd41422b0 2024-03-29T10:53:03.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 10 monclient: dump: 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cda616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cc4001640 con 0x7f7cd41422b0 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:03.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cda616640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:03.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.492+0000 7f7cda616640 4 set_mon_vals no callback set 2024-03-29T10:53:03.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.493+0000 7f7cda616640 10 monclient: shutdown 2024-03-29T10:53:03.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.493+0000 7f7cda616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:03.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.493+0000 7f7cda616640 1 -- 172.21.15.5:0/226866574 >> v1:172.21.15.177:6789/0 conn(0x7f7cd41422b0 legacy=0x7f7cd413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.493+0000 7f7cd2ffd640 1 -- 172.21.15.5:0/226866574 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7cc4001460 con 0x7f7cd41422b0 2024-03-29T10:53:03.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.494+0000 7f7cd2ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:53:03.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.497+0000 7f7cda616640 1 -- 172.21.15.5:0/226866574 shutdown_connections 2024-03-29T10:53:03.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.497+0000 7f7cda616640 1 -- 172.21.15.5:0/226866574 >> 172.21.15.5:0/226866574 conn(0x7f7cd4066a00 msgr2=0x7f7cd4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:03.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.497+0000 7f7cda616640 1 -- 172.21.15.5:0/226866574 shutdown_connections 2024-03-29T10:53:03.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.499+0000 7f7cda616640 1 -- 172.21.15.5:0/226866574 wait complete. 2024-03-29T10:53:03.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.500+0000 7f7cda616640 10 monclient: build_initial_monmap 2024-03-29T10:53:03.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.500+0000 7f7cda616640 10 monclient: monmap: 2024-03-29T10:53:03.482 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:03.482 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.500+0000 7f7cda616640 1 librados: starting msgr at 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.500+0000 7f7cda616640 1 librados: starting objecter 2024-03-29T10:53:03.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.501+0000 7f7cda616640 1 Processor -- start 2024-03-29T10:53:03.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.502+0000 7f7cda616640 1 -- start start 2024-03-29T10:53:03.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.502+0000 7f7cda616640 1 librados: setting wanted keys 2024-03-29T10:53:03.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.502+0000 7f7cda616640 1 librados: calling monclient init 2024-03-29T10:53:03.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.502+0000 7f7cda616640 10 monclient: init 2024-03-29T10:53:03.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.503+0000 7f7cda616640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:03.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.503+0000 7f7cda616640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:03.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.503+0000 7f7cda616640 10 monclient(hunting): picked mon.noname-a con 0x7f7cd40b17d0 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:03.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cda616640 10 monclient(hunting): picked mon.noname-c con 0x7f7cd41422b0 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:03.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cda616640 10 monclient(hunting): picked mon.noname-b con 0x7f7cd4142690 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:03.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cda616640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd41e7560 con 0x7f7cd40b17d0 2024-03-29T10:53:03.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cda616640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd4130cd0 con 0x7f7cd4142690 2024-03-29T10:53:03.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cda616640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd406f7d0 con 0x7f7cd41422b0 2024-03-29T10:53:03.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cda616640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:03.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cda616640 10 monclient(hunting): authenticate will time out at 1196.745193s 2024-03-29T10:53:03.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cd37fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f7cd41422b0 0x7f7cd41e79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:50960/0 (socket says 172.21.15.5:50960) 2024-03-29T10:53:03.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cd37fe640 1 -- 172.21.15.5:0/1975006707 learned_addr learned my addr 172.21.15.5:0/1975006707 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:03.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cd0ff9640 10 client.?.objecter ms_handle_connect 0x7f7cd41422b0 2024-03-29T10:53:03.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:03.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354554240 0 0) 0x7f7cd406f7d0 con 0x7f7cd41422b0 2024-03-29T10:53:03.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:03.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:03.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cd0ff9640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.504+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc0004eb0 con 0x7f7cd41422b0 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.505+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 763648530 0 0) 0x7f7cc0004eb0 con 0x7f7cd41422b0 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.505+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cd406f7d0 con 0x7f7cd41422b0 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.505+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7cc8001970 con 0x7f7cd41422b0 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.505+0000 7f7cd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.505+0000 7f7cd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.505+0000 7f7cd0ff9640 10 monclient(hunting): dump: 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:03.493 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:03.494 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:03.494 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.494 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:03.494 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:03.494 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:03.494 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:03.494 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:03.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.505+0000 7f7cd0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:03.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.505+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2862278665 0 0) 0x7f7cd406f7d0 con 0x7f7cd41422b0 2024-03-29T10:53:03.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.505+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 >> v1:172.21.15.177:6789/0 conn(0x7f7cd4142690 legacy=0x7f7cd41eeb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:03.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.507+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 >> v1:172.21.15.5:6789/0 conn(0x7f7cd40b17d0 legacy=0x7f7cd41335d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.507+0000 7f7cd0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:53:03.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.507+0000 7f7cd0ff9640 1 monclient: found mon.c 2024-03-29T10:53:03.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.507+0000 7f7cd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:03.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.507+0000 7f7cd0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:03.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.507+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd406fab0 con 0x7f7cd41422b0 2024-03-29T10:53:03.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.507+0000 7f7cd0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:53:03.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.507+0000 7f7cd0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:53:03.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.507+0000 7f7cd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:03.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cc80047e0 con 0x7f7cd41422b0 2024-03-29T10:53:03.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:03.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7cc8004c30 con 0x7f7cd41422b0 2024-03-29T10:53:03.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:03.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cd0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:03.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cd0ff9640 10 monclient: dump: 2024-03-29T10:53:03.496 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:03.496 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:03.496 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:03.496 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:03.497 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:03.497 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:03.497 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:03.497 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:03.497 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:03.497 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:03.497 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cd27fc640 4 set_mon_vals no callback set 2024-03-29T10:53:03.497 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cda616640 5 monclient: authenticate success, global_id 4463 2024-03-29T10:53:03.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cda616640 10 monclient: _renew_subs 2024-03-29T10:53:03.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cda616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:03.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cda616640 1 -- 172.21.15.5:0/1975006707 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd406e3e0 con 0x7f7cd41422b0 2024-03-29T10:53:03.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cda616640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:03.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cda616640 10 monclient: _renew_subs 2024-03-29T10:53:03.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cda616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:03.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.508+0000 7f7cda616640 1 -- 172.21.15.5:0/1975006707 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd41e82e0 con 0x7f7cd41422b0 2024-03-29T10:53:03.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.509+0000 7f7cda616640 1 librados: init done 2024-03-29T10:53:03.500 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.511+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f7cc8004d90 con 0x7f7cd41422b0 2024-03-29T10:53:03.500 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.511+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f7cc8084b90 con 0x7f7cd41422b0 2024-03-29T10:53:03.500 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.511+0000 7f7cd0ff9640 10 client.4463.objecter ms_dispatch 0x7f7cd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:03.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.511+0000 7f7cd0ff9640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:03.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.511+0000 7f7cd0ff9640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:03.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.511+0000 7f7cda616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:03.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.511+0000 7f7cda616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:03.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.511+0000 7f7cda616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:03.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.511+0000 7f7cd0ff9640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T10:53:03.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.511+0000 7f7cda616640 1 -- 172.21.15.5:0/1975006707 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd4075f10 con 0x7f7cd41422b0 2024-03-29T10:53:03.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.512+0000 7f7cd3fff640 10 monclient: get_auth_request con 0x7f7cc006dcd0 auth_method 0 2024-03-29T10:53:03.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.513+0000 7f7cd0ff9640 10 client.4463.objecter ms_handle_connect 0x7f7cc006dcd0 2024-03-29T10:53:03.502 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.520+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7cc8050470 con 0x7f7cd41422b0 2024-03-29T10:53:03.502 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.520+0000 7f7cd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:03.502 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.520+0000 7f7cd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:03.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.855+0000 7f7cda616640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:03.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.855+0000 7f7cda616640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:03.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.855+0000 7f7cda616640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:03.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.855+0000 7f7cda616640 1 -- 172.21.15.5:0/1975006707 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cd40b2270 con 0x7f7cd41422b0 2024-03-29T10:53:03.838 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.856+0000 7f7cd0ff9640 1 -- 172.21.15.5:0/1975006707 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f7cc80523a0 con 0x7f7cd41422b0 2024-03-29T10:53:03.838 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.856+0000 7f7cd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:03.838 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.856+0000 7f7cd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:03.838 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:53:03.838 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:03.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.860+0000 7f7cda616640 10 librados: watch_flush enter 2024-03-29T10:53:03.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.860+0000 7f7cda616640 10 librados: watch_flush exit 2024-03-29T10:53:03.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.860+0000 7f7cda616640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T10:53:03.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.860+0000 7f7cda616640 10 client.4463.objecter successfully canceled tick 2024-03-29T10:53:03.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.860+0000 7f7cda616640 1 -- 172.21.15.5:0/1975006707 >> v1:172.21.15.5:6816/2847500151 conn(0x7f7cc006dcd0 legacy=0x7f7cc0070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.860+0000 7f7cda616640 10 monclient: shutdown 2024-03-29T10:53:03.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.860+0000 7f7cda616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:03.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.860+0000 7f7cda616640 1 -- 172.21.15.5:0/1975006707 >> v1:172.21.15.5:6790/0 conn(0x7f7cd41422b0 legacy=0x7f7cd41e79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:03.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.860+0000 7f7cda616640 1 -- 172.21.15.5:0/1975006707 shutdown_connections 2024-03-29T10:53:03.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.861+0000 7f7cda616640 1 -- 172.21.15.5:0/1975006707 >> 172.21.15.5:0/1975006707 conn(0x7f7cd4066a00 msgr2=0x7f7cd4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:03.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.861+0000 7f7cda616640 1 -- 172.21.15.5:0/1975006707 shutdown_connections 2024-03-29T10:53:03.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.861+0000 7f7cda616640 1 -- 172.21.15.5:0/1975006707 wait complete. 2024-03-29T10:53:03.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:03.861+0000 7f7cda616640 1 librados: shutdown 2024-03-29T10:53:03.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:10.666 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:10.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.834+0000 7f391615a640 10 monclient: get_monmap_and_config 2024-03-29T10:53:10.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.834+0000 7f391615a640 10 monclient: build_initial_monmap 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.834+0000 7f391615a640 10 monclient: monmap: 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:53:10.835242+0000 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:53:10.835242+0000 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:53:10.816 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:10.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.836+0000 7f391615a640 1 Processor -- start 2024-03-29T10:53:10.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.836+0000 7f391615a640 1 -- start start 2024-03-29T10:53:10.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.836+0000 7f391615a640 10 monclient: init 2024-03-29T10:53:10.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.837+0000 7f391615a640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.837+0000 7f391615a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.837+0000 7f391615a640 10 monclient(hunting): picked mon.noname-a con 0x7f39080389e0 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.837+0000 7f391615a640 10 monclient(hunting): picked mon.noname-c con 0x7f3908038dc0 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.837+0000 7f391615a640 10 monclient(hunting): picked mon.noname-b con 0x7f3908030660 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.837+0000 7f391615a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39080c54c0 con 0x7f39080389e0 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.837+0000 7f391615a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3908036430 con 0x7f3908038dc0 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.837+0000 7f391615a640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3908039840 con 0x7f3908030660 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.837+0000 7f391615a640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.837+0000 7f391615a640 10 monclient(hunting): authenticate will time out at 1204.078581s 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.840+0000 7f3915158640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f39080389e0 0x7f3908033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43298/0 (socket says 172.21.15.5:43298) 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.840+0000 7f3915158640 1 -- 172.21.15.5:0/435747264 learned_addr learned my addr 172.21.15.5:0/435747264 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.840+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099131389 0 0) 0x7f39080c54c0 con 0x7f39080389e0 2024-03-29T10:53:10.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.840+0000 7f3907fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.840+0000 7f3907fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.840+0000 7f3907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.840+0000 7f3907fff640 10 monclient(hunting): my global_id is 4401 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.840+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38fc003520 con 0x7f39080389e0 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718051208 0 0) 0x7f3908039840 con 0x7f3908030660 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39080c54c0 con 0x7f3908030660 2024-03-29T10:53:10.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 795983664 0 0) 0x7f38fc003520 con 0x7f39080389e0 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 >> v1:172.21.15.5:6790/0 conn(0x7f3908030660 legacy=0x7f39080d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 >> v1:172.21.15.177:6789/0 conn(0x7f3908038dc0 legacy=0x7f39080cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 1 monclient: found mon.noname-a 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3908035690 con 0x7f39080389e0 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f3900001690 con 0x7f39080389e0 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.841+0000 7f3907fff640 10 monclient: dump: 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:10.827 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:10.830 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.843+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3900001920 con 0x7f39080389e0 2024-03-29T10:53:10.830 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.843+0000 7f3907fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:10.830 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.843+0000 7f3907fff640 1 -- 172.21.15.5:0/435747264 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f3900001cb0 con 0x7f39080389e0 2024-03-29T10:53:10.830 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.843+0000 7f3907fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.830 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.843+0000 7f3907fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:10.830 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.843+0000 7f3907fff640 10 monclient: dump: 2024-03-29T10:53:10.830 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:10.830 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:10.830 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:10.830 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.843+0000 7f391615a640 5 monclient: authenticate success, global_id 4401 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.843+0000 7f391615a640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.843+0000 7f391615a640 4 set_mon_vals no callback set 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.844+0000 7f391615a640 10 monclient: shutdown 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.844+0000 7f391615a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.844+0000 7f391615a640 1 -- 172.21.15.5:0/435747264 >> v1:172.21.15.5:6789/0 conn(0x7f39080389e0 legacy=0x7f3908033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.844+0000 7f391615a640 1 -- 172.21.15.5:0/435747264 shutdown_connections 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.844+0000 7f391615a640 1 -- 172.21.15.5:0/435747264 >> 172.21.15.5:0/435747264 conn(0x7f3908005c00 msgr2=0x7f3908006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:10.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.845+0000 7f391615a640 1 -- 172.21.15.5:0/435747264 shutdown_connections 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.845+0000 7f391615a640 1 -- 172.21.15.5:0/435747264 wait complete. 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.846+0000 7f391615a640 10 monclient: build_initial_monmap 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.846+0000 7f391615a640 10 monclient: monmap: 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.846+0000 7f391615a640 1 librados: starting msgr at 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.846+0000 7f391615a640 1 librados: starting objecter 2024-03-29T10:53:10.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.846+0000 7f391615a640 1 Processor -- start 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.847+0000 7f391615a640 1 -- start start 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.847+0000 7f391615a640 1 librados: setting wanted keys 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.847+0000 7f391615a640 1 librados: calling monclient init 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.847+0000 7f391615a640 10 monclient: init 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.847+0000 7f391615a640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.847+0000 7f391615a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.847+0000 7f391615a640 10 monclient(hunting): picked mon.noname-a con 0x7f3908030660 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.847+0000 7f391615a640 10 monclient(hunting): picked mon.noname-c con 0x7f39080389e0 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.848+0000 7f391615a640 10 monclient(hunting): picked mon.noname-b con 0x7f39080c67c0 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.848+0000 7f391615a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39080c5e90 con 0x7f3908030660 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.848+0000 7f391615a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39080c7ea0 con 0x7f39080c67c0 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.848+0000 7f391615a640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390800d820 con 0x7f39080389e0 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.848+0000 7f391615a640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:10.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.848+0000 7f391615a640 10 monclient(hunting): authenticate will time out at 1204.089128s 2024-03-29T10:53:10.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.849+0000 7f3915158640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3908030660 0x7f39080be270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44720/0 (socket says 172.21.15.5:44720) 2024-03-29T10:53:10.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.849+0000 7f3915158640 1 -- 172.21.15.5:0/2116681659 learned_addr learned my addr 172.21.15.5:0/2116681659 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.849+0000 7f3915158640 1 -- 172.21.15.5:0/2116681659 >> v1:172.21.15.5:6789/0 conn(0x7f3908030660 legacy=0x7f39080be270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.849+0000 7f3915158640 1 -- 172.21.15.5:0/2116681659 >> v1:172.21.15.5:6789/0 conn(0x7f3908030660 legacy=0x7f39080be270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.849+0000 7f3915158640 1 --1- 172.21.15.5:0/2116681659 >> v1:172.21.15.5:6789/0 conn(0x7f3908030660 0x7f39080be270 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.849+0000 7f3915158640 1 --1- 172.21.15.5:0/2116681659 >> v1:172.21.15.5:6789/0 conn(0x7f3908030660 0x7f39080be270 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.852+0000 7f3905ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.5:6789/0 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.853+0000 7f3905ffb640 10 client.?.objecter ms_handle_connect 0x7f39080389e0 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.853+0000 7f3905ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.853+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818333544 0 0) 0x7f390800d820 con 0x7f39080389e0 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.854+0000 7f3905ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.854+0000 7f3905ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.854+0000 7f3905ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.854+0000 7f3905ffb640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.854+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f4004eb0 con 0x7f39080389e0 2024-03-29T10:53:10.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.854+0000 7f3905ffb640 10 client.?.objecter ms_handle_connect 0x7f39080c67c0 2024-03-29T10:53:10.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.854+0000 7f3905ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:10.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848091451 0 0) 0x7f39080c7ea0 con 0x7f39080c67c0 2024-03-29T10:53:10.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:10.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f390800d820 con 0x7f39080c67c0 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1622396948 0 0) 0x7f38f4004eb0 con 0x7f39080389e0 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39080c7ea0 con 0x7f39080389e0 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f390c001750 con 0x7f39080389e0 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.858+0000 7f3905ffb640 10 monclient(hunting): dump: 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:10.843 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1376478640 0 0) 0x7f39080c7ea0 con 0x7f39080389e0 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 >> v1:172.21.15.177:6789/0 conn(0x7f39080c67c0 legacy=0x7f3908031730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 >> v1:172.21.15.5:6789/0 conn(0x7f3908030660 legacy=0x7f39080be270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 1 monclient: found mon.c 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39080c5ab0 con 0x7f39080389e0 2024-03-29T10:53:10.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f391615a640 5 monclient: authenticate success, global_id 4469 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f391615a640 10 monclient: _renew_subs 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f391615a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f391615a640 1 -- 172.21.15.5:0/2116681659 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39080c6100 con 0x7f39080389e0 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f391615a640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f391615a640 10 monclient: _renew_subs 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f390c001ce0 con 0x7f39080389e0 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f391615a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f391615a640 1 -- 172.21.15.5:0/2116681659 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39080beb60 con 0x7f39080389e0 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.859+0000 7f391615a640 1 librados: init done 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.860+0000 7f3905ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.860+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f390c003790 con 0x7f39080389e0 2024-03-29T10:53:10.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.860+0000 7f3905ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.860+0000 7f3905ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.860+0000 7f3905ffb640 10 monclient: dump: 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.861+0000 7f391615a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:10.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.861+0000 7f391615a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:10.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.861+0000 7f391615a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:10.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.861+0000 7f391615a640 1 -- 172.21.15.5:0/2116681659 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3908007b90 con 0x7f39080389e0 2024-03-29T10:53:10.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.862+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f390c0038f0 con 0x7f39080389e0 2024-03-29T10:53:10.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.862+0000 7f3906ffd640 4 set_mon_vals no callback set 2024-03-29T10:53:10.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.862+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f390c083750 con 0x7f39080389e0 2024-03-29T10:53:10.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.862+0000 7f3905ffb640 10 client.4469.objecter ms_dispatch 0x7f3910000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:10.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.862+0000 7f3905ffb640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:10.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.862+0000 7f3905ffb640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:10.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.862+0000 7f3905ffb640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T10:53:10.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.863+0000 7f3915158640 10 monclient: get_auth_request con 0x7f38f406d980 auth_method 0 2024-03-29T10:53:10.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.863+0000 7f3905ffb640 10 client.4469.objecter ms_handle_connect 0x7f38f406d980 2024-03-29T10:53:10.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.870+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f390c04ee60 con 0x7f39080389e0 2024-03-29T10:53:10.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.870+0000 7f3905ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:10.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:10.870+0000 7f3905ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:11.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.210+0000 7f391615a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:11.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.210+0000 7f391615a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:11.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.210+0000 7f391615a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:11.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.210+0000 7f391615a640 1 -- 172.21.15.5:0/2116681659 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39080c83c0 con 0x7f39080389e0 2024-03-29T10:53:11.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.212+0000 7f3905ffb640 1 -- 172.21.15.5:0/2116681659 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f39080c83c0 con 0x7f39080389e0 2024-03-29T10:53:11.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.212+0000 7f3905ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:11.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.212+0000 7f3905ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:11.194 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:53:11.194 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:11.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.215+0000 7f391615a640 10 librados: watch_flush enter 2024-03-29T10:53:11.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.215+0000 7f391615a640 10 librados: watch_flush exit 2024-03-29T10:53:11.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.215+0000 7f391615a640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T10:53:11.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.215+0000 7f391615a640 10 client.4469.objecter successfully canceled tick 2024-03-29T10:53:11.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.216+0000 7f391615a640 1 -- 172.21.15.5:0/2116681659 >> v1:172.21.15.5:6816/2847500151 conn(0x7f38f406d980 legacy=0x7f38f406fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:11.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.216+0000 7f391615a640 10 monclient: shutdown 2024-03-29T10:53:11.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.216+0000 7f391615a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:11.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.216+0000 7f391615a640 1 -- 172.21.15.5:0/2116681659 >> v1:172.21.15.5:6790/0 conn(0x7f39080389e0 legacy=0x7f3908031020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:11.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.216+0000 7f391615a640 1 -- 172.21.15.5:0/2116681659 shutdown_connections 2024-03-29T10:53:11.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.216+0000 7f391615a640 1 -- 172.21.15.5:0/2116681659 >> 172.21.15.5:0/2116681659 conn(0x7f3908005c00 msgr2=0x7f390803ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:11.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.217+0000 7f391615a640 1 -- 172.21.15.5:0/2116681659 shutdown_connections 2024-03-29T10:53:11.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.217+0000 7f391615a640 1 -- 172.21.15.5:0/2116681659 wait complete. 2024-03-29T10:53:11.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:11.217+0000 7f391615a640 1 librados: shutdown 2024-03-29T10:53:11.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:18.024 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:18.175 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.194+0000 7f24105cf640 10 monclient: get_monmap_and_config 2024-03-29T10:53:18.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.194+0000 7f24105cf640 10 monclient: build_initial_monmap 2024-03-29T10:53:18.176 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.194+0000 7f24105cf640 10 monclient: monmap: 2024-03-29T10:53:18.176 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:18.176 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:18.176 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:53:18.195441+0000 2024-03-29T10:53:18.176 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:53:18.195441+0000 2024-03-29T10:53:18.176 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:18.176 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:18.176 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:18.176 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:53:18.177 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:53:18.177 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:18.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.195+0000 7f24105cf640 1 Processor -- start 2024-03-29T10:53:18.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.195+0000 7f24105cf640 1 -- start start 2024-03-29T10:53:18.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.195+0000 7f24105cf640 10 monclient: init 2024-03-29T10:53:18.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f24105cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:18.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f24105cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:18.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f24105cf640 10 monclient(hunting): picked mon.noname-c con 0x7f24080b1f60 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:18.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f24105cf640 10 monclient(hunting): picked mon.noname-b con 0x7f2408071eb0 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:18.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f24105cf640 10 monclient(hunting): picked mon.noname-a con 0x7f2408143d10 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:18.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f24105cf640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240806ef80 con 0x7f2408143d10 2024-03-29T10:53:18.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f24105cf640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240806f260 con 0x7f24080b1f60 2024-03-29T10:53:18.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f24105cf640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24080b40c0 con 0x7f2408071eb0 2024-03-29T10:53:18.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f24105cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:18.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f24105cf640 10 monclient(hunting): authenticate will time out at 1211.437834s 2024-03-29T10:53:18.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f240eb6c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f2408143d10 0x7f2408146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38346/0 (socket says 172.21.15.5:38346) 2024-03-29T10:53:18.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.196+0000 7f240eb6c640 1 -- 172.21.15.5:0/3327505386 learned_addr learned my addr 172.21.15.5:0/3327505386 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:18.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 1 -- 172.21.15.5:0/3327505386 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680877038 0 0) 0x7f240806ef80 con 0x7f2408143d10 2024-03-29T10:53:18.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:53:18.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 1 -- 172.21.15.5:0/3327505386 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23f4003520 con 0x7f2408143d10 2024-03-29T10:53:18.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 1 -- 172.21.15.5:0/3327505386 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3293842112 0 0) 0x7f23f4003520 con 0x7f2408143d10 2024-03-29T10:53:18.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 1 -- 172.21.15.5:0/3327505386 >> v1:172.21.15.5:6790/0 conn(0x7f2408071eb0 legacy=0x7f2408146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 1 -- 172.21.15.5:0/3327505386 >> v1:172.21.15.177:6789/0 conn(0x7f24080b1f60 legacy=0x7f2408143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient: _finish_hunting 0 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 1 monclient: found mon.noname-a 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 1 -- 172.21.15.5:0/3327505386 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2408155d70 con 0x7f2408143d10 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient: _finish_auth 0 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient: _check_auth_tickets 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 1 -- 172.21.15.5:0/3327505386 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2400001a10 con 0x7f2408143d10 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:18.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f240d369640 10 monclient: dump: 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f24105cf640 5 monclient: authenticate success, global_id 4404 2024-03-29T10:53:18.182 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.197+0000 7f24105cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.198+0000 7f240d369640 1 -- 172.21.15.5:0/3327505386 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2400001b70 con 0x7f2408143d10 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.198+0000 7f240d369640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.198+0000 7f240d369640 1 -- 172.21.15.5:0/3327505386 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2400001f80 con 0x7f2408143d10 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.198+0000 7f240d369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.198+0000 7f240d369640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.198+0000 7f240d369640 10 monclient: dump: 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:18.183 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:18.184 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:18.184 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:18.184 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:18.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.198+0000 7f24105cf640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:18.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.198+0000 7f24105cf640 4 set_mon_vals no callback set 2024-03-29T10:53:18.190 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.199+0000 7f24105cf640 10 monclient: shutdown 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.199+0000 7f24105cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.199+0000 7f24105cf640 1 -- 172.21.15.5:0/3327505386 >> v1:172.21.15.5:6789/0 conn(0x7f2408143d10 legacy=0x7f2408146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.199+0000 7f24105cf640 1 -- 172.21.15.5:0/3327505386 shutdown_connections 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.199+0000 7f24105cf640 1 -- 172.21.15.5:0/3327505386 >> 172.21.15.5:0/3327505386 conn(0x7f2408066a00 msgr2=0x7f2408066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.199+0000 7f24105cf640 1 -- 172.21.15.5:0/3327505386 shutdown_connections 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.199+0000 7f24105cf640 1 -- 172.21.15.5:0/3327505386 wait complete. 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.200+0000 7f24105cf640 10 monclient: build_initial_monmap 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.200+0000 7f24105cf640 10 monclient: monmap: 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:18.191 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.200+0000 7f24105cf640 1 librados: starting msgr at 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.200+0000 7f24105cf640 1 librados: starting objecter 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.201+0000 7f24105cf640 1 Processor -- start 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.201+0000 7f24105cf640 1 -- start start 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.201+0000 7f24105cf640 1 librados: setting wanted keys 2024-03-29T10:53:18.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.201+0000 7f24105cf640 1 librados: calling monclient init 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.201+0000 7f24105cf640 10 monclient: init 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.201+0000 7f24105cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.201+0000 7f24105cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f24105cf640 10 monclient(hunting): picked mon.noname-a con 0x7f24080b1f60 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f24105cf640 10 monclient(hunting): picked mon.noname-b con 0x7f2408143d10 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f24105cf640 10 monclient(hunting): picked mon.noname-c con 0x7f2408149380 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f24105cf640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240813bbc0 con 0x7f24080b1f60 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f24105cf640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2408149840 con 0x7f2408143d10 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f24105cf640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240813b940 con 0x7f2408149380 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f24105cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:18.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f24105cf640 10 monclient(hunting): authenticate will time out at 1211.443591s 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f240e36b640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f24080b1f60 0x7f240813f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38970/0 (socket says 172.21.15.5:38970) 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f240e36b640 1 -- 172.21.15.5:0/10739894 learned_addr learned my addr 172.21.15.5:0/10739894 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f23faffd640 10 client.?.objecter ms_handle_connect 0x7f2408149380 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f23faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f23faffd640 10 client.?.objecter ms_handle_connect 0x7f24080b1f60 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.202+0000 7f23faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 client.?.objecter ms_handle_connect 0x7f2408143d10 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621098591 0 0) 0x7f240806f260 con 0x7f24080b1f60 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ec004eb0 con 0x7f24080b1f60 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410943016 0 0) 0x7f2408149840 con 0x7f2408143d10 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240806f260 con 0x7f2408143d10 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850088430 0 0) 0x7f240813b940 con 0x7f2408149380 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:18.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.203+0000 7f23faffd640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:53:18.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2408149840 con 0x7f2408149380 2024-03-29T10:53:18.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2386560554 0 0) 0x7f240806f260 con 0x7f2408143d10 2024-03-29T10:53:18.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f240813b940 con 0x7f2408143d10 2024-03-29T10:53:18.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f24040018b0 con 0x7f2408143d10 2024-03-29T10:53:18.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:18.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient(hunting): dump: 2024-03-29T10:53:18.198 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:18.198 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:18.198 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:18.198 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:18.198 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.198 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:18.198 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:18.199 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:18.199 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:18.199 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:18.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:18.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2064179292 0 0) 0x7f23ec004eb0 con 0x7f24080b1f60 2024-03-29T10:53:18.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f240806f260 con 0x7f24080b1f60 2024-03-29T10:53:18.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f23fc001810 con 0x7f24080b1f60 2024-03-29T10:53:18.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:18.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient(hunting): dump: 2024-03-29T10:53:18.200 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:18.200 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:18.200 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:18.200 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:18.200 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.200 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:18.200 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3497442521 0 0) 0x7f240806f260 con 0x7f24080b1f60 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 >> v1:172.21.15.5:6790/0 conn(0x7f2408149380 legacy=0x7f240813b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 >> v1:172.21.15.177:6789/0 conn(0x7f2408143d10 legacy=0x7f240813ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 monclient: found mon.a 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:18.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240814af40 con 0x7f24080b1f60 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f23faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f24105cf640 5 monclient: authenticate success, global_id 4407 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f24105cf640 10 monclient: _renew_subs 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f24105cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f24105cf640 1 -- 172.21.15.5:0/10739894 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240814b220 con 0x7f24080b1f60 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f24105cf640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f24105cf640 10 monclient: _renew_subs 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f24105cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:18.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.204+0000 7f24105cf640 1 -- 172.21.15.5:0/10739894 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2408070980 con 0x7f24080b1f60 2024-03-29T10:53:18.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.205+0000 7f24105cf640 1 librados: init done 2024-03-29T10:53:18.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.205+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23fc001d00 con 0x7f24080b1f60 2024-03-29T10:53:18.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.205+0000 7f23faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:18.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.205+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f23fc0037b0 con 0x7f24080b1f60 2024-03-29T10:53:18.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.205+0000 7f23faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:18.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.205+0000 7f23fbfff640 4 set_mon_vals no callback set 2024-03-29T10:53:18.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.205+0000 7f23faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:18.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.205+0000 7f23faffd640 10 monclient: dump: 2024-03-29T10:53:18.203 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.207+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f23fc003910 con 0x7f24080b1f60 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.207+0000 7f24105cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:18.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.207+0000 7f24105cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:18.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.207+0000 7f24105cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:18.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.207+0000 7f24105cf640 1 -- 172.21.15.5:0/10739894 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2408075f10 con 0x7f24080b1f60 2024-03-29T10:53:18.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.208+0000 7f240db6a640 10 monclient: get_auth_request con 0x7f23ec06e000 auth_method 0 2024-03-29T10:53:18.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.208+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f23fc0848e0 con 0x7f24080b1f60 2024-03-29T10:53:18.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.208+0000 7f23faffd640 10 client.4407.objecter ms_dispatch 0x7f2408000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:18.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.208+0000 7f23faffd640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:18.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.208+0000 7f23faffd640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:18.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.208+0000 7f23faffd640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T10:53:18.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.208+0000 7f23faffd640 10 client.4407.objecter ms_handle_connect 0x7f23ec06e000 2024-03-29T10:53:18.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.216+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23fc0501c0 con 0x7f24080b1f60 2024-03-29T10:53:18.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.216+0000 7f23faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:18.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.216+0000 7f23faffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:18.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.553+0000 7f24105cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:18.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.553+0000 7f24105cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:18.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.553+0000 7f24105cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:18.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.553+0000 7f24105cf640 1 -- 172.21.15.5:0/10739894 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2408074f50 con 0x7f24080b1f60 2024-03-29T10:53:18.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.554+0000 7f23faffd640 1 -- 172.21.15.5:0/10739894 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f23fc0520f0 con 0x7f24080b1f60 2024-03-29T10:53:18.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.554+0000 7f23faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:18.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.554+0000 7f23faffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:18.537 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:53:18.537 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:18.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.557+0000 7f24105cf640 10 librados: watch_flush enter 2024-03-29T10:53:18.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.557+0000 7f24105cf640 10 librados: watch_flush exit 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.557+0000 7f24105cf640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.557+0000 7f24105cf640 10 client.4407.objecter successfully canceled tick 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.557+0000 7f24105cf640 1 -- 172.21.15.5:0/10739894 >> v1:172.21.15.5:6816/2847500151 conn(0x7f23ec06e000 legacy=0x7f23ec0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.557+0000 7f24105cf640 10 monclient: shutdown 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.558+0000 7f24105cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.558+0000 7f24105cf640 1 -- 172.21.15.5:0/10739894 >> v1:172.21.15.5:6789/0 conn(0x7f24080b1f60 legacy=0x7f240813f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.558+0000 7f24105cf640 1 -- 172.21.15.5:0/10739894 shutdown_connections 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.558+0000 7f24105cf640 1 -- 172.21.15.5:0/10739894 >> 172.21.15.5:0/10739894 conn(0x7f2408066a00 msgr2=0x7f24080b7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.558+0000 7f24105cf640 1 -- 172.21.15.5:0/10739894 shutdown_connections 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.558+0000 7f24105cf640 1 -- 172.21.15.5:0/10739894 wait complete. 2024-03-29T10:53:18.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:18.558+0000 7f24105cf640 1 librados: shutdown 2024-03-29T10:53:18.555 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:25.362 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:25.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.528+0000 7f69345c8640 10 monclient: get_monmap_and_config 2024-03-29T10:53:25.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.528+0000 7f69345c8640 10 monclient: build_initial_monmap 2024-03-29T10:53:25.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.528+0000 7f69345c8640 10 monclient: monmap: 2024-03-29T10:53:25.510 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:25.510 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:25.510 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:53:25.529773+0000 2024-03-29T10:53:25.510 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:53:25.529773+0000 2024-03-29T10:53:25.510 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:25.511 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:25.511 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:25.511 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:53:25.511 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:53:25.511 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:25.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.530+0000 7f69345c8640 1 Processor -- start 2024-03-29T10:53:25.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.530+0000 7f69345c8640 1 -- start start 2024-03-29T10:53:25.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.530+0000 7f69345c8640 10 monclient: init 2024-03-29T10:53:25.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.531+0000 7f69345c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:25.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.531+0000 7f69345c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:25.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.532+0000 7f69345c8640 10 monclient(hunting): picked mon.noname-a con 0x7f692c1457f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:25.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.532+0000 7f69345c8640 10 monclient(hunting): picked mon.noname-c con 0x7f692c13f650 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:25.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.532+0000 7f69345c8640 10 monclient(hunting): picked mon.noname-b con 0x7f692c0a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:25.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.532+0000 7f6932364640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f692c1457f0 0x7f692c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35430/0 (socket says 172.21.15.5:35430) 2024-03-29T10:53:25.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.532+0000 7f6932364640 1 -- 172.21.15.5:0/118207804 learned_addr learned my addr 172.21.15.5:0/118207804 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:25.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.532+0000 7f69345c8640 1 -- 172.21.15.5:0/118207804 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c0aaef0 con 0x7f692c1457f0 2024-03-29T10:53:25.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f69345c8640 1 -- 172.21.15.5:0/118207804 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c145bd0 con 0x7f692c13f650 2024-03-29T10:53:25.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f6931b63640 1 -- 172.21.15.5:0/118207804 >> v1:172.21.15.177:6789/0 conn(0x7f692c13f650 legacy=0x7f692c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-29T10:53:25.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f6931b63640 1 -- 172.21.15.5:0/118207804 >> v1:172.21.15.177:6789/0 conn(0x7f692c13f650 legacy=0x7f692c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T10:53:25.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f6931b63640 1 --1- 172.21.15.5:0/118207804 >> v1:172.21.15.177:6789/0 conn(0x7f692c13f650 0x7f692c0b2620 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-29T10:53:25.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f69345c8640 1 -- 172.21.15.5:0/118207804 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c145eb0 con 0x7f692c0a9d70 2024-03-29T10:53:25.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f69345c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:25.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f69345c8640 10 monclient(hunting): authenticate will time out at 1218.774257s 2024-03-29T10:53:25.516 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f6931362640 1 -- 172.21.15.5:0/118207804 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842950221 0 0) 0x7f692c0aaef0 con 0x7f692c1457f0 2024-03-29T10:53:25.516 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f6931362640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.516 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f6931362640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.516 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f6931362640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.516 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.533+0000 7f6931362640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:53:25.516 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.534+0000 7f6931362640 1 -- 172.21.15.5:0/118207804 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6918003520 con 0x7f692c1457f0 2024-03-29T10:53:25.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 1 -- 172.21.15.5:0/118207804 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 450890994 0 0) 0x7f6918003520 con 0x7f692c1457f0 2024-03-29T10:53:25.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 1 -- 172.21.15.5:0/118207804 >> v1:172.21.15.5:6790/0 conn(0x7f692c0a9d70 legacy=0x7f692c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 1 -- 172.21.15.5:0/118207804 >> v1:172.21.15.177:6789/0 conn(0x7f692c13f650 legacy=0x7f692c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: _finish_hunting 0 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 1 monclient: found mon.noname-a 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 1 -- 172.21.15.5:0/118207804 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c146340 con 0x7f692c1457f0 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: _finish_auth 0 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: _check_auth_tickets 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 1 -- 172.21.15.5:0/118207804 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6928001690 con 0x7f692c1457f0 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: dump: 2024-03-29T10:53:25.518 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f69345c8640 5 monclient: authenticate success, global_id 4410 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f69345c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 1 -- 172.21.15.5:0/118207804 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6928001920 con 0x7f692c1457f0 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:25.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 1 -- 172.21.15.5:0/118207804 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6928001cb0 con 0x7f692c1457f0 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f6931362640 10 monclient: dump: 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f69345c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.535+0000 7f69345c8640 4 set_mon_vals no callback set 2024-03-29T10:53:25.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.536+0000 7f69345c8640 10 monclient: shutdown 2024-03-29T10:53:25.521 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.536+0000 7f69345c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:25.521 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.536+0000 7f69345c8640 1 -- 172.21.15.5:0/118207804 >> v1:172.21.15.5:6789/0 conn(0x7f692c1457f0 legacy=0x7f692c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.527 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.537+0000 7f69345c8640 1 -- 172.21.15.5:0/118207804 shutdown_connections 2024-03-29T10:53:25.527 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.537+0000 7f69345c8640 1 -- 172.21.15.5:0/118207804 >> 172.21.15.5:0/118207804 conn(0x7f692c066a00 msgr2=0x7f692c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:25.528 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.537+0000 7f69345c8640 1 -- 172.21.15.5:0/118207804 shutdown_connections 2024-03-29T10:53:25.528 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.537+0000 7f69345c8640 1 -- 172.21.15.5:0/118207804 wait complete. 2024-03-29T10:53:25.528 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.538+0000 7f69345c8640 10 monclient: build_initial_monmap 2024-03-29T10:53:25.528 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.538+0000 7f69345c8640 10 monclient: monmap: 2024-03-29T10:53:25.528 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:25.528 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:25.528 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:53:25.528 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:53:25.528 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.539+0000 7f69345c8640 1 librados: starting msgr at 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.539+0000 7f69345c8640 1 librados: starting objecter 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.539+0000 7f69345c8640 1 Processor -- start 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.539+0000 7f69345c8640 1 -- start start 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.539+0000 7f69345c8640 1 librados: setting wanted keys 2024-03-29T10:53:25.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.539+0000 7f69345c8640 1 librados: calling monclient init 2024-03-29T10:53:25.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.539+0000 7f69345c8640 10 monclient: init 2024-03-29T10:53:25.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.541+0000 7f69345c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:25.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.541+0000 7f69345c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:25.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.541+0000 7f69345c8640 10 monclient(hunting): picked mon.noname-a con 0x7f692c0a9d70 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:25.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.541+0000 7f69345c8640 10 monclient(hunting): picked mon.noname-c con 0x7f692c13f650 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:25.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.541+0000 7f69345c8640 10 monclient(hunting): picked mon.noname-b con 0x7f692c1457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:25.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.541+0000 7f69345c8640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c142020 con 0x7f692c0a9d70 2024-03-29T10:53:25.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.541+0000 7f6932b65640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f692c1457f0 0x7f692c1e2af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:50796/0 (socket says 172.21.15.5:50796) 2024-03-29T10:53:25.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.541+0000 7f6932364640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f692c0a9d70 0x7f692c1425e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35616/0 (socket says 172.21.15.5:35616) 2024-03-29T10:53:25.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.541+0000 7f6932364640 1 -- 172.21.15.5:0/582685586 learned_addr learned my addr 172.21.15.5:0/582685586 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:25.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.542+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c135a20 con 0x7f692c1457f0 2024-03-29T10:53:25.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.542+0000 7f6922ffd640 10 client.?.objecter ms_handle_connect 0x7f692c1457f0 2024-03-29T10:53:25.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.542+0000 7f6922ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:25.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.542+0000 7f6922ffd640 10 client.?.objecter ms_handle_connect 0x7f692c0a9d70 2024-03-29T10:53:25.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.542+0000 7f6922ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:25.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.542+0000 7f6922ffd640 10 client.?.objecter ms_handle_connect 0x7f692c13f650 2024-03-29T10:53:25.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.542+0000 7f6922ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:25.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.542+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c135d00 con 0x7f692c13f650 2024-03-29T10:53:25.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.542+0000 7f69345c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:25.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577408634 0 0) 0x7f692c135a20 con 0x7f692c1457f0 2024-03-29T10:53:25.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f69345c8640 10 monclient(hunting): authenticate will time out at 1218.784250s 2024-03-29T10:53:25.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:53:25.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6910004eb0 con 0x7f692c1457f0 2024-03-29T10:53:25.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914147422 0 0) 0x7f692c142020 con 0x7f692c0a9d70 2024-03-29T10:53:25.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:53:25.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692c135a20 con 0x7f692c0a9d70 2024-03-29T10:53:25.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728835735 0 0) 0x7f692c135d00 con 0x7f692c13f650 2024-03-29T10:53:25.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:25.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:25.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:25.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:53:25.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.543+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692c142020 con 0x7f692c13f650 2024-03-29T10:53:25.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1168761913 0 0) 0x7f692c135a20 con 0x7f692c0a9d70 2024-03-29T10:53:25.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f692c135d00 con 0x7f692c0a9d70 2024-03-29T10:53:25.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3328275055 0 0) 0x7f6910004eb0 con 0x7f692c1457f0 2024-03-29T10:53:25.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6910006a90 con 0x7f692c1457f0 2024-03-29T10:53:25.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6928001550 con 0x7f692c0a9d70 2024-03-29T10:53:25.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:25.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 10 monclient(hunting): dump: 2024-03-29T10:53:25.535 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:25.535 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:25.535 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.535 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.535 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.535 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:25.535 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:25.535 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6924001c70 con 0x7f692c1457f0 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 10 monclient(hunting): dump: 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.536 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.537 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.537 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:25.537 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:25.537 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:25.537 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:25.537 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:25.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3691842732 0 0) 0x7f692c142020 con 0x7f692c13f650 2024-03-29T10:53:25.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69100050f0 con 0x7f692c13f650 2024-03-29T10:53:25.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f691c003f70 con 0x7f692c13f650 2024-03-29T10:53:25.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.544+0000 7f6922ffd640 10 monclient(hunting): dump: 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:25.538 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:25.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2404355374 0 0) 0x7f692c135d00 con 0x7f692c0a9d70 2024-03-29T10:53:25.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 >> v1:172.21.15.5:6790/0 conn(0x7f692c13f650 legacy=0x7f692c133d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 >> v1:172.21.15.177:6789/0 conn(0x7f692c1457f0 legacy=0x7f692c1e2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:25.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:25.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 1 monclient: found mon.a 2024-03-29T10:53:25.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:25.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:25.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c141cb0 con 0x7f692c0a9d70 2024-03-29T10:53:25.539 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f69345c8640 5 monclient: authenticate success, global_id 4413 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f69345c8640 10 monclient: _renew_subs 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f69345c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c142cf0 con 0x7f692c0a9d70 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f69345c8640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f69345c8640 10 monclient: _renew_subs 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f69345c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c13ff80 con 0x7f692c0a9d70 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f69345c8640 1 librados: init done 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.545+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6928004a90 con 0x7f692c0a9d70 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.546+0000 7f6922ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:25.540 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.546+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f6928004da0 con 0x7f692c0a9d70 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.546+0000 7f6922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.546+0000 7f6922ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.546+0000 7f6922ffd640 10 monclient: dump: 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:25.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.546+0000 7f6930b61640 4 set_mon_vals no callback set 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.547+0000 7f69345c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.547+0000 7f69345c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.547+0000 7f69345c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.547+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6914005080 con 0x7f692c0a9d70 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.548+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f692801ad70 con 0x7f692c0a9d70 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.549+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f6928084a40 con 0x7f692c0a9d70 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.549+0000 7f6922ffd640 10 client.4413.objecter ms_dispatch 0x7f692c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.549+0000 7f6922ffd640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.549+0000 7f6922ffd640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.549+0000 7f6922ffd640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.549+0000 7f6931b63640 10 monclient: get_auth_request con 0x7f691006e640 auth_method 0 2024-03-29T10:53:25.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.550+0000 7f6922ffd640 10 client.4413.objecter ms_handle_connect 0x7f691006e640 2024-03-29T10:53:25.544 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.559+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6928050320 con 0x7f692c0a9d70 2024-03-29T10:53:25.544 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.559+0000 7f6922ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:25.544 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.559+0000 7f6922ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:25.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.932+0000 7f69345c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:25.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.932+0000 7f69345c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:25.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.932+0000 7f69345c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:25.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.932+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6914005330 con 0x7f692c0a9d70 2024-03-29T10:53:25.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.933+0000 7f6922ffd640 1 -- 172.21.15.5:0/582685586 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f6928052250 con 0x7f692c0a9d70 2024-03-29T10:53:25.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.933+0000 7f6922ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:25.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.933+0000 7f6922ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:25.918 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:53:26.531 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:26.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.937+0000 7f69345c8640 10 librados: watch_flush enter 2024-03-29T10:53:26.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.937+0000 7f69345c8640 10 librados: watch_flush exit 2024-03-29T10:53:26.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.937+0000 7f69345c8640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-29T10:53:26.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.937+0000 7f69345c8640 10 client.4413.objecter successfully canceled tick 2024-03-29T10:53:26.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.937+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 >> v1:172.21.15.5:6816/2847500151 conn(0x7f691006e640 legacy=0x7f6910070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:26.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.937+0000 7f69345c8640 10 monclient: shutdown 2024-03-29T10:53:26.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.937+0000 7f69345c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:26.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.937+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 >> v1:172.21.15.5:6789/0 conn(0x7f692c0a9d70 legacy=0x7f692c1425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:26.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.938+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 shutdown_connections 2024-03-29T10:53:26.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.938+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 >> 172.21.15.5:0/582685586 conn(0x7f692c066a00 msgr2=0x7f692c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:26.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.938+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 shutdown_connections 2024-03-29T10:53:26.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.938+0000 7f69345c8640 1 -- 172.21.15.5:0/582685586 wait complete. 2024-03-29T10:53:26.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:25.938+0000 7f69345c8640 1 librados: shutdown 2024-03-29T10:53:26.534 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:33.340 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:33.472 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.491+0000 7f463508d640 10 monclient: get_monmap_and_config 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.491+0000 7f463508d640 10 monclient: build_initial_monmap 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.491+0000 7f463508d640 10 monclient: monmap: 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:53:33.492579+0000 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:53:33.492579+0000 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:53:33.473 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:33.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.492+0000 7f463508d640 1 Processor -- start 2024-03-29T10:53:33.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.492+0000 7f463508d640 1 -- start start 2024-03-29T10:53:33.474 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.492+0000 7f463508d640 10 monclient: init 2024-03-29T10:53:33.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f463508d640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:33.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f463508d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:33.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f463508d640 10 monclient(hunting): picked mon.noname-a con 0x7f46301457f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:33.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f463508d640 10 monclient(hunting): picked mon.noname-c con 0x7f463013f650 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:33.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f463508d640 10 monclient(hunting): picked mon.noname-b con 0x7f46300a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f463508d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46300aaef0 con 0x7f46301457f0 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f463508d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4630145bd0 con 0x7f463013f650 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f463508d640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4630145eb0 con 0x7f46300a9d70 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f463508d640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f463508d640 10 monclient(hunting): authenticate will time out at 1226.734619s 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f462e59c640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f463013f650 0x7f46300b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:46764/0 (socket says 172.21.15.5:46764) 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.493+0000 7f462e59c640 1 -- 172.21.15.5:0/1681459880 learned_addr learned my addr 172.21.15.5:0/1681459880 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 1 -- 172.21.15.5:0/1681459880 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791961174 0 0) 0x7f4630145bd0 con 0x7f463013f650 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 1 -- 172.21.15.5:0/1681459880 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f461800bd30 con 0x7f463013f650 2024-03-29T10:53:33.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 1 -- 172.21.15.5:0/1681459880 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192820250 0 0) 0x7f46300aaef0 con 0x7f46301457f0 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 1 -- 172.21.15.5:0/1681459880 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f461800d0b0 con 0x7f46301457f0 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 1 -- 172.21.15.5:0/1681459880 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4175639453 0 0) 0x7f461800d0b0 con 0x7f46301457f0 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.494+0000 7f462dd9b640 1 -- 172.21.15.5:0/1681459880 >> v1:172.21.15.5:6790/0 conn(0x7f46300a9d70 legacy=0x7f463014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 1 -- 172.21.15.5:0/1681459880 >> v1:172.21.15.177:6789/0 conn(0x7f463013f650 legacy=0x7f46300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 1 monclient: found mon.noname-a 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 1 -- 172.21.15.5:0/1681459880 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4630146340 con 0x7f46301457f0 2024-03-29T10:53:33.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 10 monclient: _finish_auth 0 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 1 -- 172.21.15.5:0/1681459880 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4620001670 con 0x7f46301457f0 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 10 monclient: dump: 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:33.481 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:33.482 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:33.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f463508d640 5 monclient: authenticate success, global_id 4416 2024-03-29T10:53:33.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f463508d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:33.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 1 -- 172.21.15.5:0/1681459880 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4620001850 con 0x7f46301457f0 2024-03-29T10:53:33.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f462dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:33.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f463508d640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:33.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.496+0000 7f463508d640 4 set_mon_vals no callback set 2024-03-29T10:53:33.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.497+0000 7f463508d640 10 monclient: shutdown 2024-03-29T10:53:33.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.497+0000 7f463508d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:33.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.497+0000 7f463508d640 1 -- 172.21.15.5:0/1681459880 >> v1:172.21.15.5:6789/0 conn(0x7f46301457f0 legacy=0x7f463013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.498+0000 7f463508d640 1 -- 172.21.15.5:0/1681459880 shutdown_connections 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.498+0000 7f463508d640 1 -- 172.21.15.5:0/1681459880 >> 172.21.15.5:0/1681459880 conn(0x7f4630066a00 msgr2=0x7f4630066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.498+0000 7f463508d640 1 -- 172.21.15.5:0/1681459880 shutdown_connections 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.498+0000 7f463508d640 1 -- 172.21.15.5:0/1681459880 wait complete. 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.499+0000 7f463508d640 10 monclient: build_initial_monmap 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.499+0000 7f463508d640 10 monclient: monmap: 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:33.484 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.499+0000 7f463508d640 1 librados: starting msgr at 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.499+0000 7f463508d640 1 librados: starting objecter 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.500+0000 7f463508d640 1 Processor -- start 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.500+0000 7f463508d640 1 -- start start 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.500+0000 7f463508d640 1 librados: setting wanted keys 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.500+0000 7f463508d640 1 librados: calling monclient init 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.500+0000 7f463508d640 10 monclient: init 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.501+0000 7f463508d640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.501+0000 7f463508d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.501+0000 7f463508d640 10 monclient(hunting): picked mon.noname-a con 0x7f46300a9d70 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.501+0000 7f462ed9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f46300a9d70 0x7f463013e3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35786/0 (socket says 172.21.15.5:35786) 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.501+0000 7f462ed9d640 1 -- 172.21.15.5:0/1732906343 learned_addr learned my addr 172.21.15.5:0/1732906343 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:33.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.501+0000 7f46177fe640 10 client.?.objecter ms_handle_connect 0x7f46300a9d70 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.501+0000 7f46177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.502+0000 7f463508d640 10 monclient(hunting): picked mon.noname-c con 0x7f463013f650 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.502+0000 7f463508d640 10 monclient(hunting): picked mon.noname-b con 0x7f46301457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.502+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4630144590 con 0x7f46300a9d70 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.502+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4630145220 con 0x7f46301457f0 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.502+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46301328d0 con 0x7f463013f650 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.502+0000 7f463508d640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.502+0000 7f463508d640 10 monclient(hunting): authenticate will time out at 1226.743740s 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797361190 0 0) 0x7f4630144590 con 0x7f46300a9d70 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:53:33.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f461c004eb0 con 0x7f46300a9d70 2024-03-29T10:53:33.490 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 client.?.objecter ms_handle_connect 0x7f46301457f0 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 26931972 0 0) 0x7f461c004eb0 con 0x7f46300a9d70 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4630144590 con 0x7f46300a9d70 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4620003850 con 0x7f46300a9d70 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): dump: 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:33.491 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302279765 0 0) 0x7f4630145220 con 0x7f46301457f0 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.503+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f461c004eb0 con 0x7f46301457f0 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3648125699 0 0) 0x7f4630144590 con 0x7f46300a9d70 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 >> v1:172.21.15.5:6790/0 conn(0x7f463013f650 legacy=0x7f46301321c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 >> v1:172.21.15.177:6789/0 conn(0x7f46301457f0 legacy=0x7f46301d2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 10 monclient: _finish_hunting 0 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 1 monclient: found mon.a 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4630144c90 con 0x7f46300a9d70 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 10 monclient: _finish_auth 0 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 10 monclient: _check_auth_tickets 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f463508d640 5 monclient: authenticate success, global_id 4419 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.505+0000 7f463508d640 10 monclient: _renew_subs 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.505+0000 7f463508d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.505+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4630144ee0 con 0x7f46300a9d70 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.505+0000 7f463508d640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.505+0000 7f463508d640 10 monclient: _renew_subs 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.505+0000 7f463508d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:33.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.505+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4630143380 con 0x7f46300a9d70 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.505+0000 7f463508d640 1 librados: init done 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.504+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4620003da0 con 0x7f46300a9d70 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.505+0000 7f46177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.505+0000 7f462d59a640 4 set_mon_vals no callback set 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.506+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f46200041f0 con 0x7f46300a9d70 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.506+0000 7f46177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.506+0000 7f46177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.506+0000 7f46177fe640 10 monclient: dump: 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:33.494 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.507+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f4620001b00 con 0x7f46300a9d70 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.507+0000 7f463508d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.507+0000 7f463508d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.507+0000 7f463508d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.507+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4610005080 con 0x7f46300a9d70 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.507+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f4620084cd0 con 0x7f46300a9d70 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.508+0000 7f46177fe640 10 client.4419.objecter ms_dispatch 0x7f4630000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.508+0000 7f46177fe640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.508+0000 7f46177fe640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.508+0000 7f46177fe640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.508+0000 7f462e59c640 10 monclient: get_auth_request con 0x7f461c06db60 auth_method 0 2024-03-29T10:53:33.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.508+0000 7f46177fe640 10 client.4419.objecter ms_handle_connect 0x7f461c06db60 2024-03-29T10:53:33.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.515+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4620050610 con 0x7f46300a9d70 2024-03-29T10:53:33.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.515+0000 7f46177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:33.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.515+0000 7f46177fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:33.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.879+0000 7f463508d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:33.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.879+0000 7f463508d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:33.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.879+0000 7f463508d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:53:33.861 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.879+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4610005330 con 0x7f46300a9d70 2024-03-29T10:53:33.862 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.880+0000 7f46177fe640 1 -- 172.21.15.5:0/1732906343 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f4620084380 con 0x7f46300a9d70 2024-03-29T10:53:33.862 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.880+0000 7f46177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:33.862 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.880+0000 7f46177fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:33.862 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:53:33.862 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:33.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.883+0000 7f463508d640 10 librados: watch_flush enter 2024-03-29T10:53:33.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.883+0000 7f463508d640 10 librados: watch_flush exit 2024-03-29T10:53:33.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.883+0000 7f463508d640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-29T10:53:33.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.883+0000 7f463508d640 10 client.4419.objecter successfully canceled tick 2024-03-29T10:53:33.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.883+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 >> v1:172.21.15.5:6816/2847500151 conn(0x7f461c06db60 legacy=0x7f461c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.883+0000 7f463508d640 10 monclient: shutdown 2024-03-29T10:53:33.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.883+0000 7f463508d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:33.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.883+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 >> v1:172.21.15.5:6789/0 conn(0x7f46300a9d70 legacy=0x7f463013e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:33.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.884+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 shutdown_connections 2024-03-29T10:53:33.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.884+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 >> 172.21.15.5:0/1732906343 conn(0x7f4630066a00 msgr2=0x7f463013a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:33.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.884+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 shutdown_connections 2024-03-29T10:53:33.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.884+0000 7f463508d640 1 -- 172.21.15.5:0/1732906343 wait complete. 2024-03-29T10:53:33.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:33.884+0000 7f463508d640 1 librados: shutdown 2024-03-29T10:53:33.879 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:40.685 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:40.841 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.860+0000 7efcfd425640 10 monclient: get_monmap_and_config 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.860+0000 7efcfd425640 10 monclient: build_initial_monmap 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.860+0000 7efcfd425640 10 monclient: monmap: 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:53:40.861596+0000 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:53:40.861596+0000 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:53:40.842 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:40.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.864+0000 7efcfd425640 1 Processor -- start 2024-03-29T10:53:40.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.864+0000 7efcfd425640 1 -- start start 2024-03-29T10:53:40.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.864+0000 7efcfd425640 10 monclient: init 2024-03-29T10:53:40.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.865+0000 7efcfd425640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:40.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.865+0000 7efcfd425640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:40.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.865+0000 7efcfd425640 10 monclient(hunting): picked mon.noname-b con 0x7efcf81457f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:40.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.865+0000 7efcfd425640 10 monclient(hunting): picked mon.noname-a con 0x7efcf813f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:40.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.865+0000 7efcfd425640 10 monclient(hunting): picked mon.noname-c con 0x7efcf80a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:40.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.865+0000 7efcfd425640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf80aaef0 con 0x7efcf813f650 2024-03-29T10:53:40.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.865+0000 7efcfd425640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf8145bd0 con 0x7efcf80a9d70 2024-03-29T10:53:40.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.865+0000 7efcfd425640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf8145eb0 con 0x7efcf81457f0 2024-03-29T10:53:40.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.865+0000 7efcfd425640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:40.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.865+0000 7efcfd425640 10 monclient(hunting): authenticate will time out at 1234.106894s 2024-03-29T10:53:40.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.869+0000 7efcf77fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7efcf80a9d70 0x7efcf814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:54158/0 (socket says 172.21.15.5:54158) 2024-03-29T10:53:40.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.869+0000 7efcf77fe640 1 -- 172.21.15.5:0/3102030592 learned_addr learned my addr 172.21.15.5:0/3102030592 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:40.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343815352 0 0) 0x7efcf8145bd0 con 0x7efcf80a9d70 2024-03-29T10:53:40.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:53:40.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce4003520 con 0x7efcf80a9d70 2024-03-29T10:53:40.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380431705 0 0) 0x7efcf80aaef0 con 0x7efcf813f650 2024-03-29T10:53:40.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:53:40.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf8145bd0 con 0x7efcf813f650 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165092062 0 0) 0x7efcf8145bd0 con 0x7efcf813f650 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 >> v1:172.21.15.5:6790/0 conn(0x7efcf81457f0 legacy=0x7efcf813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 >> v1:172.21.15.177:6789/0 conn(0x7efcf80a9d70 legacy=0x7efcf814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 1 monclient: found mon.noname-a 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf8146340 con 0x7efcf813f650 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.870+0000 7efcf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:40.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efce0001670 con 0x7efcf813f650 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcf67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcf67fc640 10 monclient: dump: 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:40.855 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efce0001850 con 0x7efcf813f650 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcf67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcf67fc640 1 -- 172.21.15.5:0/3102030592 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efce0001670 con 0x7efcf813f650 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcf67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcf67fc640 10 monclient: dump: 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:40.856 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:40.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcfd425640 5 monclient: authenticate success, global_id 4422 2024-03-29T10:53:40.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcfd425640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:40.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.871+0000 7efcfd425640 4 set_mon_vals no callback set 2024-03-29T10:53:40.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.872+0000 7efcfd425640 10 monclient: shutdown 2024-03-29T10:53:40.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.872+0000 7efcfd425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:40.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.872+0000 7efcfd425640 1 -- 172.21.15.5:0/3102030592 >> v1:172.21.15.5:6789/0 conn(0x7efcf813f650 legacy=0x7efcf80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.872+0000 7efcfd425640 1 -- 172.21.15.5:0/3102030592 shutdown_connections 2024-03-29T10:53:40.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.872+0000 7efcfd425640 1 -- 172.21.15.5:0/3102030592 >> 172.21.15.5:0/3102030592 conn(0x7efcf8066a00 msgr2=0x7efcf8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:40.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.873+0000 7efcfd425640 1 -- 172.21.15.5:0/3102030592 shutdown_connections 2024-03-29T10:53:40.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.873+0000 7efcfd425640 1 -- 172.21.15.5:0/3102030592 wait complete. 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.874+0000 7efcfd425640 10 monclient: build_initial_monmap 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.874+0000 7efcfd425640 10 monclient: monmap: 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:40.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.874+0000 7efcfd425640 1 librados: starting msgr at 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.874+0000 7efcfd425640 1 librados: starting objecter 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.875+0000 7efcfd425640 1 Processor -- start 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.875+0000 7efcfd425640 1 -- start start 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.875+0000 7efcfd425640 1 librados: setting wanted keys 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.875+0000 7efcfd425640 1 librados: calling monclient init 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.875+0000 7efcfd425640 10 monclient: init 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.876+0000 7efcfd425640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.876+0000 7efcfd425640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.877+0000 7efcfd425640 10 monclient(hunting): picked mon.noname-c con 0x7efcf80a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.877+0000 7efcfd425640 10 monclient(hunting): picked mon.noname-b con 0x7efcf813f650 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.877+0000 7efcfd425640 10 monclient(hunting): picked mon.noname-a con 0x7efcf81457f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.877+0000 7efcfd425640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf80acdb0 con 0x7efcf81457f0 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.877+0000 7efcfd425640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf81dadc0 con 0x7efcf813f650 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.877+0000 7efcfd425640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf81da9e0 con 0x7efcf80a9d70 2024-03-29T10:53:40.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.877+0000 7efcfd425640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:40.861 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.877+0000 7efcfd425640 10 monclient(hunting): authenticate will time out at 1234.118350s 2024-03-29T10:53:40.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.878+0000 7efcee7fc640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7efcf813f650 0x7efcf8137410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:55034/0 (socket says 172.21.15.5:55034) 2024-03-29T10:53:40.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.878+0000 7efcee7fc640 1 -- 172.21.15.5:0/665458512 learned_addr learned my addr 172.21.15.5:0/665458512 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:40.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.879+0000 7efceffff640 10 client.?.objecter ms_handle_connect 0x7efcf813f650 2024-03-29T10:53:40.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.879+0000 7efceffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:40.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.879+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806070339 0 0) 0x7efcf81dadc0 con 0x7efcf813f650 2024-03-29T10:53:40.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.879+0000 7efceffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.879+0000 7efceffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.879+0000 7efceffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.879+0000 7efceffff640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.879+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd8004eb0 con 0x7efcf813f650 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.879+0000 7efceffff640 10 client.?.objecter ms_handle_connect 0x7efcf80a9d70 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.879+0000 7efceffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577926069 0 0) 0x7efcf81da9e0 con 0x7efcf80a9d70 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf81dadc0 con 0x7efcf80a9d70 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 849073735 0 0) 0x7efcd8004eb0 con 0x7efcf813f650 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcf81da9e0 con 0x7efcf813f650 2024-03-29T10:53:40.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efce0001670 con 0x7efcf813f650 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): dump: 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.865 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545762809 0 0) 0x7efcf81dadc0 con 0x7efcf80a9d70 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcd8004eb0 con 0x7efcf80a9d70 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efce8001ac0 con 0x7efcf80a9d70 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.880+0000 7efceffff640 10 monclient(hunting): dump: 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:40.866 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:40.867 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 374617787 0 0) 0x7efcd8004eb0 con 0x7efcf80a9d70 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 >> v1:172.21.15.177:6789/0 conn(0x7efcf813f650 legacy=0x7efcf8137410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 >> v1:172.21.15.5:6789/0 conn(0x7efcf81457f0 legacy=0x7efcf81e2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 1 monclient: found mon.c 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf81dab70 con 0x7efcf80a9d70 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 10 monclient: _finish_auth 0 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efce8001f70 con 0x7efcf80a9d70 2024-03-29T10:53:40.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efce8003b30 con 0x7efcf80a9d70 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efceffff640 10 monclient: dump: 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:40.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.881+0000 7efcf57fa640 4 set_mon_vals no callback set 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.882+0000 7efcfd425640 5 monclient: authenticate success, global_id 4487 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.882+0000 7efcfd425640 10 monclient: _renew_subs 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.882+0000 7efcfd425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.882+0000 7efcfd425640 1 -- 172.21.15.5:0/665458512 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf81420a0 con 0x7efcf80a9d70 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.882+0000 7efcfd425640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.882+0000 7efcfd425640 10 monclient: _renew_subs 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.882+0000 7efcfd425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.882+0000 7efcfd425640 1 -- 172.21.15.5:0/665458512 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf8142e50 con 0x7efcf80a9d70 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.882+0000 7efcfd425640 1 librados: init done 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.884+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7efce8003c90 con 0x7efcf80a9d70 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.885+0000 7efcfd425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.885+0000 7efcfd425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:40.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.885+0000 7efcfd425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.885+0000 7efcfd425640 1 -- 172.21.15.5:0/665458512 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf81db000 con 0x7efcf80a9d70 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.885+0000 7efcee7fc640 10 monclient: get_auth_request con 0x7efcd806df60 auth_method 0 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.886+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7efce8083860 con 0x7efcf80a9d70 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.886+0000 7efceffff640 10 client.4487.objecter ms_dispatch 0x7efcf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.886+0000 7efceffff640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.886+0000 7efceffff640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.886+0000 7efceffff640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T10:53:40.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.886+0000 7efceffff640 10 client.4487.objecter ms_handle_connect 0x7efcd806df60 2024-03-29T10:53:40.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.894+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efcf81db000 con 0x7efcf80a9d70 2024-03-29T10:53:40.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.894+0000 7efceffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:40.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:40.895+0000 7efceffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:41.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.238+0000 7efcfd425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:41.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.238+0000 7efcfd425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:41.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.238+0000 7efcfd425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:41.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.238+0000 7efcfd425640 1 -- 172.21.15.5:0/665458512 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcf8142e50 con 0x7efcf80a9d70 2024-03-29T10:53:41.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.239+0000 7efceffff640 1 -- 172.21.15.5:0/665458512 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7efcf8142e50 con 0x7efcf80a9d70 2024-03-29T10:53:41.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.239+0000 7efceffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:41.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.239+0000 7efceffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:41.221 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:53:41.222 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:41.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.243+0000 7efcfd425640 10 librados: watch_flush enter 2024-03-29T10:53:41.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.243+0000 7efcfd425640 10 librados: watch_flush exit 2024-03-29T10:53:41.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.243+0000 7efcfd425640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T10:53:41.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.243+0000 7efcfd425640 10 client.4487.objecter successfully canceled tick 2024-03-29T10:53:41.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.244+0000 7efcfd425640 1 -- 172.21.15.5:0/665458512 >> v1:172.21.15.5:6816/2847500151 conn(0x7efcd806df60 legacy=0x7efcd8070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:41.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.244+0000 7efcfd425640 10 monclient: shutdown 2024-03-29T10:53:41.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.244+0000 7efcfd425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:41.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.244+0000 7efcfd425640 1 -- 172.21.15.5:0/665458512 >> v1:172.21.15.5:6790/0 conn(0x7efcf80a9d70 legacy=0x7efcf81db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:41.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.245+0000 7efcfd425640 1 -- 172.21.15.5:0/665458512 shutdown_connections 2024-03-29T10:53:41.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.245+0000 7efcfd425640 1 -- 172.21.15.5:0/665458512 >> 172.21.15.5:0/665458512 conn(0x7efcf8066a00 msgr2=0x7efcf813aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:41.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.245+0000 7efcfd425640 1 -- 172.21.15.5:0/665458512 shutdown_connections 2024-03-29T10:53:41.227 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.246+0000 7efcfd425640 1 -- 172.21.15.5:0/665458512 wait complete. 2024-03-29T10:53:41.227 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:41.246+0000 7efcfd425640 1 librados: shutdown 2024-03-29T10:53:41.243 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:48.051 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:48.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.218+0000 7fe94f6c1640 10 monclient: get_monmap_and_config 2024-03-29T10:53:48.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.218+0000 7fe94f6c1640 10 monclient: build_initial_monmap 2024-03-29T10:53:48.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.218+0000 7fe94f6c1640 10 monclient: monmap: 2024-03-29T10:53:48.201 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:48.201 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:48.201 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:53:48.219593+0000 2024-03-29T10:53:48.201 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:53:48.219593+0000 2024-03-29T10:53:48.201 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:48.202 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:48.202 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:48.202 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:53:48.202 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:53:48.202 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:48.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.220+0000 7fe94f6c1640 1 Processor -- start 2024-03-29T10:53:48.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.220+0000 7fe94f6c1640 1 -- start start 2024-03-29T10:53:48.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.220+0000 7fe94f6c1640 10 monclient: init 2024-03-29T10:53:48.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.221+0000 7fe94f6c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:48.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.221+0000 7fe94f6c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:48.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.221+0000 7fe94f6c1640 10 monclient(hunting): picked mon.noname-c con 0x7fe9480a9ac0 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:48.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.221+0000 7fe94f6c1640 10 monclient(hunting): picked mon.noname-a con 0x7fe94813ee90 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:48.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.221+0000 7fe94f6c1640 10 monclient(hunting): picked mon.noname-b con 0x7fe9480af900 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:48.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.221+0000 7fe94f6c1640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9480afd10 con 0x7fe94813ee90 2024-03-29T10:53:48.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.222+0000 7fe94dc5e640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fe9480af900 0x7fe948149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:33548/0 (socket says 172.21.15.5:33548) 2024-03-29T10:53:48.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.222+0000 7fe94dc5e640 1 -- 172.21.15.5:0/2913789422 learned_addr learned my addr 172.21.15.5:0/2913789422 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:48.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.222+0000 7fe94f6c1640 1 -- 172.21.15.5:0/2913789422 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9480af6f0 con 0x7fe9480a9ac0 2024-03-29T10:53:48.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.222+0000 7fe94f6c1640 1 -- 172.21.15.5:0/2913789422 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9480aac80 con 0x7fe9480af900 2024-03-29T10:53:48.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.222+0000 7fe94f6c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:48.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.222+0000 7fe94f6c1640 10 monclient(hunting): authenticate will time out at 1241.463344s 2024-03-29T10:53:48.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384623183 0 0) 0x7fe9480aac80 con 0x7fe9480af900 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe934003520 con 0x7fe9480af900 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905625674 0 0) 0x7fe9480af6f0 con 0x7fe9480a9ac0 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9480aac80 con 0x7fe9480a9ac0 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1478213820 0 0) 0x7fe934003520 con 0x7fe9480af900 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 >> v1:172.21.15.177:6789/0 conn(0x7fe9480a9ac0 legacy=0x7fe9480adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.224+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 >> v1:172.21.15.5:6789/0 conn(0x7fe94813ee90 legacy=0x7fe94813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 1 monclient: found mon.noname-b 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9480aaf60 con 0x7fe9480af900 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 10 monclient: _finish_auth 0 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe938001690 con 0x7fe9480af900 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 10 monclient: dump: 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:48.212 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe94f6c1640 5 monclient: authenticate success, global_id 4490 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe94f6c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9380018b0 con 0x7fe9480af900 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe93ffff640 1 -- 172.21.15.5:0/2913789422 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe938001c40 con 0x7fe9480af900 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe94f6c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:48.213 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.229+0000 7fe94f6c1640 4 set_mon_vals no callback set 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.230+0000 7fe94f6c1640 10 monclient: shutdown 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.230+0000 7fe93ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.230+0000 7fe93ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.230+0000 7fe93ffff640 10 monclient: dump: 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:48.214 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.230+0000 7fe94f6c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:48.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.230+0000 7fe94f6c1640 1 -- 172.21.15.5:0/2913789422 >> v1:172.21.15.5:6790/0 conn(0x7fe9480af900 legacy=0x7fe948149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.231+0000 7fe94f6c1640 1 -- 172.21.15.5:0/2913789422 shutdown_connections 2024-03-29T10:53:48.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.231+0000 7fe94f6c1640 1 -- 172.21.15.5:0/2913789422 >> 172.21.15.5:0/2913789422 conn(0x7fe9480b35e0 msgr2=0x7fe9480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:48.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.231+0000 7fe94f6c1640 1 -- 172.21.15.5:0/2913789422 shutdown_connections 2024-03-29T10:53:48.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.231+0000 7fe94f6c1640 1 -- 172.21.15.5:0/2913789422 wait complete. 2024-03-29T10:53:48.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.232+0000 7fe94f6c1640 10 monclient: build_initial_monmap 2024-03-29T10:53:48.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.232+0000 7fe94f6c1640 10 monclient: monmap: 2024-03-29T10:53:48.216 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:48.216 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:48.216 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:53:48.216 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:53:48.216 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:48.216 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.233+0000 7fe94f6c1640 1 librados: starting msgr at 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.233+0000 7fe94f6c1640 1 librados: starting objecter 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.233+0000 7fe94f6c1640 1 Processor -- start 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.233+0000 7fe94f6c1640 1 -- start start 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.233+0000 7fe94f6c1640 1 librados: setting wanted keys 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.233+0000 7fe94f6c1640 1 librados: calling monclient init 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.233+0000 7fe94f6c1640 10 monclient: init 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.234+0000 7fe94f6c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.234+0000 7fe94f6c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.234+0000 7fe94f6c1640 10 monclient(hunting): picked mon.noname-c con 0x7fe9480a9ac0 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:48.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.234+0000 7fe94d45d640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fe9480a9ac0 0x7fe9481da750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:37536/0 (socket says 172.21.15.5:37536) 2024-03-29T10:53:48.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.234+0000 7fe94d45d640 1 -- 172.21.15.5:0/3093053185 learned_addr learned my addr 172.21.15.5:0/3093053185 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:48.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.234+0000 7fe93dffb640 10 client.?.objecter ms_handle_connect 0x7fe9480a9ac0 2024-03-29T10:53:48.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.234+0000 7fe93dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe94f6c1640 10 monclient(hunting): picked mon.noname-b con 0x7fe9480af900 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe94f6c1640 10 monclient(hunting): picked mon.noname-a con 0x7fe94813ee90 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94813fb00 con 0x7fe94813ee90 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe948139580 con 0x7fe9480af900 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9481dae60 con 0x7fe9480a9ac0 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe94f6c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe94f6c1640 10 monclient(hunting): authenticate will time out at 1241.477270s 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572804432 0 0) 0x7fe9481dae60 con 0x7fe9480a9ac0 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe93dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe93dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe93dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe93dffb640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.236+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c004eb0 con 0x7fe9480a9ac0 2024-03-29T10:53:48.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.237+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908467556 0 0) 0x7fe92c004eb0 con 0x7fe9480a9ac0 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.237+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9481dae60 con 0x7fe9480a9ac0 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.237+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe944001a10 con 0x7fe9480a9ac0 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.237+0000 7fe93dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.237+0000 7fe93dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.237+0000 7fe93dffb640 10 monclient(hunting): dump: 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:48.221 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:48.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.237+0000 7fe93dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:48.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.237+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3782387629 0 0) 0x7fe9481dae60 con 0x7fe9480a9ac0 2024-03-29T10:53:48.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.237+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 >> v1:172.21.15.177:6789/0 conn(0x7fe9480af900 legacy=0x7fe948138760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:48.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.237+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 >> v1:172.21.15.5:6789/0 conn(0x7fe94813ee90 legacy=0x7fe948138e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:48.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe93dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:48.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe93dffb640 1 monclient: found mon.c 2024-03-29T10:53:48.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe93dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:48.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe93dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:48.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe948140c00 con 0x7fe9480a9ac0 2024-03-29T10:53:48.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe93dffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:48.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe93dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:48.224 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe93dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:48.224 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe94f6c1640 5 monclient: authenticate success, global_id 4493 2024-03-29T10:53:48.224 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe94f6c1640 10 monclient: _renew_subs 2024-03-29T10:53:48.224 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe94f6c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:48.224 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9481c9560 con 0x7fe9480a9ac0 2024-03-29T10:53:48.224 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe94f6c1640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:48.224 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe94f6c1640 10 monclient: _renew_subs 2024-03-29T10:53:48.224 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe94f6c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:48.224 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94813f5c0 con 0x7fe9480a9ac0 2024-03-29T10:53:48.224 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.239+0000 7fe94f6c1640 1 librados: init done 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.241+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9440038c0 con 0x7fe9480a9ac0 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.241+0000 7fe93dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.241+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe944003d10 con 0x7fe9480a9ac0 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.241+0000 7fe93dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.241+0000 7fe93dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.241+0000 7fe93dffb640 10 monclient: dump: 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:48.225 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:48.226 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:48.226 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:48.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.241+0000 7fe93effd640 4 set_mon_vals no callback set 2024-03-29T10:53:48.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.242+0000 7fe94f6c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:48.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.242+0000 7fe94f6c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:48.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.242+0000 7fe94f6c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:48.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.242+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe928005080 con 0x7fe9480a9ac0 2024-03-29T10:53:48.227 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.244+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fe944003e70 con 0x7fe9480a9ac0 2024-03-29T10:53:48.227 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.244+0000 7fe94cc5c640 10 monclient: get_auth_request con 0x7fe92c06dca0 auth_method 0 2024-03-29T10:53:48.227 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.244+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fe944084c60 con 0x7fe9480a9ac0 2024-03-29T10:53:48.227 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.244+0000 7fe93dffb640 10 client.4493.objecter ms_dispatch 0x7fe948000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:48.228 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.246+0000 7fe93dffb640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:48.228 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.246+0000 7fe93dffb640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:48.228 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.246+0000 7fe93dffb640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T10:53:48.228 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.246+0000 7fe93dffb640 10 client.4493.objecter ms_handle_connect 0x7fe92c06dca0 2024-03-29T10:53:48.237 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.256+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe944050540 con 0x7fe9480a9ac0 2024-03-29T10:53:48.237 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.256+0000 7fe93dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:48.237 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.256+0000 7fe93dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:48.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.579+0000 7fe94f6c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:48.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.579+0000 7fe94f6c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:48.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.579+0000 7fe94f6c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:48.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.579+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe928005330 con 0x7fe9480a9ac0 2024-03-29T10:53:48.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.581+0000 7fe93dffb640 1 -- 172.21.15.5:0/3093053185 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fe944052470 con 0x7fe9480a9ac0 2024-03-29T10:53:48.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.581+0000 7fe93dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:48.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.581+0000 7fe93dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:48.563 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:53:48.563 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:48.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.584+0000 7fe94f6c1640 10 librados: watch_flush enter 2024-03-29T10:53:48.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.585+0000 7fe94f6c1640 10 librados: watch_flush exit 2024-03-29T10:53:48.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.585+0000 7fe94f6c1640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T10:53:48.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.585+0000 7fe94f6c1640 10 client.4493.objecter successfully canceled tick 2024-03-29T10:53:48.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.586+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 >> v1:172.21.15.5:6816/2847500151 conn(0x7fe92c06dca0 legacy=0x7fe92c070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.586+0000 7fe94f6c1640 10 monclient: shutdown 2024-03-29T10:53:48.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.586+0000 7fe94f6c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:48.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.586+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 >> v1:172.21.15.5:6790/0 conn(0x7fe9480a9ac0 legacy=0x7fe9481da750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.586+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 shutdown_connections 2024-03-29T10:53:48.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.586+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 >> 172.21.15.5:0/3093053185 conn(0x7fe9480b35e0 msgr2=0x7fe9480b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:48.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.587+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 shutdown_connections 2024-03-29T10:53:48.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.587+0000 7fe94f6c1640 1 -- 172.21.15.5:0/3093053185 wait complete. 2024-03-29T10:53:48.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:48.587+0000 7fe94f6c1640 1 librados: shutdown 2024-03-29T10:53:48.585 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:55.393 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:55.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.560+0000 7f43dfa86640 10 monclient: get_monmap_and_config 2024-03-29T10:53:55.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.560+0000 7f43dfa86640 10 monclient: build_initial_monmap 2024-03-29T10:53:55.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.560+0000 7f43dfa86640 10 monclient: monmap: 2024-03-29T10:53:55.541 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:55.541 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:55.541 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:53:55.561201+0000 2024-03-29T10:53:55.541 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:53:55.561201+0000 2024-03-29T10:53:55.542 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:55.542 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:55.542 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:55.542 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:53:55.542 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:53:55.542 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:55.543 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.562+0000 7f43dfa86640 1 Processor -- start 2024-03-29T10:53:55.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.564+0000 7f43dfa86640 1 -- start start 2024-03-29T10:53:55.545 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.564+0000 7f43dfa86640 10 monclient: init 2024-03-29T10:53:55.546 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.565+0000 7f43dfa86640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:55.546 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.565+0000 7f43dfa86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:55.546 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.565+0000 7f43dfa86640 10 monclient(hunting): picked mon.noname-a con 0x7f43d0006e50 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:55.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.565+0000 7f43dfa86640 10 monclient(hunting): picked mon.noname-c con 0x7f43d0035e20 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:55.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.566+0000 7f43dfa86640 10 monclient(hunting): picked mon.noname-b con 0x7f43d00c9690 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:55.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.566+0000 7f43dfa86640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d0035990 con 0x7f43d0006e50 2024-03-29T10:53:55.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.566+0000 7f43dfa86640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d00362a0 con 0x7f43d0035e20 2024-03-29T10:53:55.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.566+0000 7f43dfa86640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d0035660 con 0x7f43d00c9690 2024-03-29T10:53:55.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.566+0000 7f43dfa86640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:55.547 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.566+0000 7f43dfa86640 10 monclient(hunting): authenticate will time out at 1248.807148s 2024-03-29T10:53:55.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.568+0000 7f43de023640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f43d00c9690 0x7f43d00d7580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:36210/0 (socket says 172.21.15.5:36210) 2024-03-29T10:53:55.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.568+0000 7f43de023640 1 -- 172.21.15.5:0/658674522 learned_addr learned my addr 172.21.15.5:0/658674522 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:55.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.569+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229686452 0 0) 0x7f43d0035990 con 0x7f43d0006e50 2024-03-29T10:53:55.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.569+0000 7f43dc820640 10 monclient(hunting): _negotiate 2024-03-29T10:53:55.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.569+0000 7f43dc820640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:55.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.569+0000 7f43dc820640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:55.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.569+0000 7f43dc820640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:53:55.550 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.569+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c4003520 con 0x7f43d0006e50 2024-03-29T10:53:55.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430606341 0 0) 0x7f43d0035660 con 0x7f43d00c9690 2024-03-29T10:53:55.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient(hunting): _negotiate 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d0035990 con 0x7f43d00c9690 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4265137259 0 0) 0x7f43c4003520 con 0x7f43d0006e50 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 >> v1:172.21.15.5:6790/0 conn(0x7f43d00c9690 legacy=0x7f43d00d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 >> v1:172.21.15.177:6789/0 conn(0x7f43d0035e20 legacy=0x7f43d00c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: _finish_hunting 0 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 1 monclient: found mon.noname-a 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d0034e40 con 0x7f43d0006e50 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: _finish_auth 0 2024-03-29T10:53:55.556 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: _check_auth_tickets 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dfa86640 5 monclient: authenticate success, global_id 4428 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dfa86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f43d4001610 con 0x7f43d0006e50 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: dump: 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:55.557 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dfa86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43d40018a0 con 0x7f43d0006e50 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dfa86640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dfa86640 4 set_mon_vals no callback set 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 1 -- 172.21.15.5:0/658674522 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f43d4001c30 con 0x7f43d0006e50 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.573+0000 7f43dc820640 10 monclient: dump: 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:55.558 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.574+0000 7f43dfa86640 10 monclient: shutdown 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.574+0000 7f43dfa86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.574+0000 7f43dfa86640 1 -- 172.21.15.5:0/658674522 >> v1:172.21.15.5:6789/0 conn(0x7f43d0006e50 legacy=0x7f43d00cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.574+0000 7f43dfa86640 1 -- 172.21.15.5:0/658674522 shutdown_connections 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.574+0000 7f43dfa86640 1 -- 172.21.15.5:0/658674522 >> 172.21.15.5:0/658674522 conn(0x7f43d0005c20 msgr2=0x7f43d0006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.574+0000 7f43dfa86640 1 -- 172.21.15.5:0/658674522 shutdown_connections 2024-03-29T10:53:55.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.574+0000 7f43dfa86640 1 -- 172.21.15.5:0/658674522 wait complete. 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.575+0000 7f43dfa86640 10 monclient: build_initial_monmap 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.575+0000 7f43dfa86640 10 monclient: monmap: 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:53:55.562 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.576+0000 7f43dfa86640 1 librados: starting msgr at 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.576+0000 7f43dfa86640 1 librados: starting objecter 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.576+0000 7f43dfa86640 1 Processor -- start 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.576+0000 7f43dfa86640 1 -- start start 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.576+0000 7f43dfa86640 1 librados: setting wanted keys 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.576+0000 7f43dfa86640 1 librados: calling monclient init 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.577+0000 7f43dfa86640 10 monclient: init 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.577+0000 7f43dfa86640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.577+0000 7f43dfa86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.578+0000 7f43dfa86640 10 monclient(hunting): picked mon.noname-b con 0x7f43d0006e50 addr v1:172.21.15.177:6789/0 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.578+0000 7f43dfa86640 10 monclient(hunting): picked mon.noname-c con 0x7f43d0035e20 addr v1:172.21.15.5:6790/0 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.578+0000 7f43dfa86640 10 monclient(hunting): picked mon.noname-a con 0x7f43d00c9690 addr v1:172.21.15.5:6789/0 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.578+0000 7f43dfa86640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d0032b80 con 0x7f43d00c9690 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.578+0000 7f43dfa86640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d0033bf0 con 0x7f43d0006e50 2024-03-29T10:53:55.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.578+0000 7f43dfa86640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d0033d80 con 0x7f43d0035e20 2024-03-29T10:53:55.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.578+0000 7f43dfa86640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:55.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.578+0000 7f43dfa86640 10 monclient(hunting): authenticate will time out at 1248.819890s 2024-03-29T10:53:55.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.579+0000 7f43dd021640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f43d0035e20 0x7f43d002f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:37532/0 (socket says 172.21.15.5:37532) 2024-03-29T10:53:55.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.579+0000 7f43dd021640 1 -- 172.21.15.5:0/3568826160 learned_addr learned my addr 172.21.15.5:0/3568826160 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.581+0000 7f43ce7fc640 10 client.?.objecter ms_handle_connect 0x7f43d0035e20 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.581+0000 7f43ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.581+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223977335 0 0) 0x7f43d0033d80 con 0x7f43d0035e20 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.581+0000 7f43ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.581+0000 7f43ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.581+0000 7f43ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.581+0000 7f43ce7fc640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.581+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43bc004eb0 con 0x7f43d0035e20 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.581+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3124824471 0 0) 0x7f43bc004eb0 con 0x7f43d0035e20 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.581+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43d0033d80 con 0x7f43d0035e20 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f43c8004080 con 0x7f43d0035e20 2024-03-29T10:53:55.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 10 monclient(hunting): dump: 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:55.570 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2299880176 0 0) 0x7f43d0033d80 con 0x7f43d0035e20 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 >> v1:172.21.15.177:6789/0 conn(0x7f43d0006e50 legacy=0x7f43d0160c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 >> v1:172.21.15.5:6789/0 conn(0x7f43d00c9690 legacy=0x7f43d016c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 1 monclient: found mon.c 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d0032730 con 0x7f43d0035e20 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.582+0000 7f43ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.583+0000 7f43dfa86640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.583+0000 7f43dfa86640 10 monclient: _renew_subs 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.583+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43c80043a0 con 0x7f43d0035e20 2024-03-29T10:53:55.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.583+0000 7f43dfa86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.583+0000 7f43dfa86640 1 -- 172.21.15.5:0/3568826160 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d0033290 con 0x7f43d0035e20 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.583+0000 7f43ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.583+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f43c8004080 con 0x7f43d0035e20 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.583+0000 7f43ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.583+0000 7f43ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.583+0000 7f43ce7fc640 10 monclient: dump: 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:53:55.572 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.584+0000 7f43cffff640 4 set_mon_vals no callback set 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.585+0000 7f43dfa86640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.585+0000 7f43dfa86640 10 monclient: _renew_subs 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.585+0000 7f43dfa86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.585+0000 7f43dfa86640 1 -- 172.21.15.5:0/3568826160 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d00cb0d0 con 0x7f43d0035e20 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.585+0000 7f43dfa86640 1 librados: init done 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.586+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f43c8001180 con 0x7f43d0035e20 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.587+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f43c8054650 con 0x7f43d0035e20 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.587+0000 7f43ce7fc640 10 client.4499.objecter ms_dispatch 0x7f43d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.587+0000 7f43ce7fc640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.587+0000 7f43ce7fc640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.587+0000 7f43ce7fc640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T10:53:55.573 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.587+0000 7f43dfa86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:55.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.587+0000 7f43dfa86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:55.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.587+0000 7f43dfa86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:55.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.587+0000 7f43dfa86640 1 -- 172.21.15.5:0/3568826160 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d0007bb0 con 0x7f43d0035e20 2024-03-29T10:53:55.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.590+0000 7f43dd822640 10 monclient: get_auth_request con 0x7f43bc06ddc0 auth_method 0 2024-03-29T10:53:55.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.590+0000 7f43ce7fc640 10 client.4499.objecter ms_handle_connect 0x7f43bc06ddc0 2024-03-29T10:53:55.577 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.595+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43c8050160 con 0x7f43d0035e20 2024-03-29T10:53:55.577 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.595+0000 7f43ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:55.577 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.595+0000 7f43ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:55.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.933+0000 7f43dfa86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:55.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.933+0000 7f43dfa86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:55.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.933+0000 7f43dfa86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:53:55.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.933+0000 7f43dfa86640 1 -- 172.21.15.5:0/3568826160 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43d0033420 con 0x7f43d0035e20 2024-03-29T10:53:55.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.934+0000 7f43ce7fc640 1 -- 172.21.15.5:0/3568826160 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f43c8052090 con 0x7f43d0035e20 2024-03-29T10:53:55.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.934+0000 7f43ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:55.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.934+0000 7f43ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:55.916 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:53:55.916 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:55.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.938+0000 7f43dfa86640 10 librados: watch_flush enter 2024-03-29T10:53:55.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.938+0000 7f43dfa86640 10 librados: watch_flush exit 2024-03-29T10:53:55.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.938+0000 7f43dfa86640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T10:53:55.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.938+0000 7f43dfa86640 10 client.4499.objecter successfully canceled tick 2024-03-29T10:53:55.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.938+0000 7f43dfa86640 1 -- 172.21.15.5:0/3568826160 >> v1:172.21.15.5:6816/2847500151 conn(0x7f43bc06ddc0 legacy=0x7f43bc070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.938+0000 7f43dfa86640 10 monclient: shutdown 2024-03-29T10:53:55.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.938+0000 7f43dfa86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:55.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.938+0000 7f43dfa86640 1 -- 172.21.15.5:0/3568826160 >> v1:172.21.15.5:6790/0 conn(0x7f43d0035e20 legacy=0x7f43d002f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:55.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.939+0000 7f43dfa86640 1 -- 172.21.15.5:0/3568826160 shutdown_connections 2024-03-29T10:53:55.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.939+0000 7f43dfa86640 1 -- 172.21.15.5:0/3568826160 >> 172.21.15.5:0/3568826160 conn(0x7f43d0005c20 msgr2=0x7f43d00c8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:55.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.939+0000 7f43dfa86640 1 -- 172.21.15.5:0/3568826160 shutdown_connections 2024-03-29T10:53:55.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.939+0000 7f43dfa86640 1 -- 172.21.15.5:0/3568826160 wait complete. 2024-03-29T10:53:55.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:53:55.939+0000 7f43dfa86640 1 librados: shutdown 2024-03-29T10:53:55.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:02.745 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:02.896 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.913+0000 7f2ca7a89640 10 monclient: get_monmap_and_config 2024-03-29T10:54:02.896 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.913+0000 7f2ca7a89640 10 monclient: build_initial_monmap 2024-03-29T10:54:02.896 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.913+0000 7f2ca7a89640 10 monclient: monmap: 2024-03-29T10:54:02.896 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:02.896 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:02.896 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:54:02.914850+0000 2024-03-29T10:54:02.897 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:54:02.914850+0000 2024-03-29T10:54:02.897 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:02.897 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:02.897 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:02.897 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:54:02.897 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:54:02.897 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:02.897 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.914+0000 7f2ca7a89640 1 Processor -- start 2024-03-29T10:54:02.898 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.915+0000 7f2ca7a89640 1 -- start start 2024-03-29T10:54:02.898 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.915+0000 7f2ca7a89640 10 monclient: init 2024-03-29T10:54:02.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.916+0000 7f2ca7a89640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:02.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.916+0000 7f2ca7a89640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:02.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.916+0000 7f2ca7a89640 10 monclient(hunting): picked mon.noname-b con 0x7f2ca00ae250 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:02.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.916+0000 7f2ca7a89640 10 monclient(hunting): picked mon.noname-a con 0x7f2ca013f970 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:02.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.916+0000 7f2ca7a89640 10 monclient(hunting): picked mon.noname-c con 0x7f2ca0142800 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:02.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.916+0000 7f2ca7a89640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca00afe00 con 0x7f2ca013f970 2024-03-29T10:54:02.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.916+0000 7f2ca7a89640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca0142c10 con 0x7f2ca0142800 2024-03-29T10:54:02.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.916+0000 7f2ca7a89640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca00afb20 con 0x7f2ca00ae250 2024-03-29T10:54:02.900 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.916+0000 7f2ca7a89640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:02.900 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.916+0000 7f2ca7a89640 10 monclient(hunting): authenticate will time out at 1256.157379s 2024-03-29T10:54:02.901 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.919+0000 7f2ca5825640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f2ca00ae250 0x7f2ca013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:48020/0 (socket says 172.21.15.5:48020) 2024-03-29T10:54:02.901 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.919+0000 7f2ca5825640 1 -- 172.21.15.5:0/2675956734 learned_addr learned my addr 172.21.15.5:0/2675956734 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:02.902 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.919+0000 7f2ca6026640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f2ca0142800 0x7f2ca014da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:42229/0 (socket says 172.21.15.5:42229) 2024-03-29T10:54:02.902 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.919+0000 7f2ca4823640 1 -- 172.21.15.5:0/2675956734 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135418490 0 0) 0x7f2ca00afb20 con 0x7f2ca00ae250 2024-03-29T10:54:02.902 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.919+0000 7f2ca4823640 10 monclient(hunting): _negotiate 2024-03-29T10:54:02.902 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.919+0000 7f2ca4823640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:02.902 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.919+0000 7f2ca4823640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:02.902 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.919+0000 7f2ca4823640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:54:02.902 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.919+0000 7f2ca4823640 1 -- 172.21.15.5:0/2675956734 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c8c003520 con 0x7f2ca00ae250 2024-03-29T10:54:02.902 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.920+0000 7f2ca4823640 1 -- 172.21.15.5:0/2675956734 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1168233573 0 0) 0x7f2c8c003520 con 0x7f2ca00ae250 2024-03-29T10:54:02.902 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.920+0000 7f2ca4823640 1 -- 172.21.15.5:0/2675956734 >> v1:172.21.15.177:6789/0 conn(0x7f2ca0142800 legacy=0x7f2ca014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.902 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.920+0000 7f2ca4823640 1 -- 172.21.15.5:0/2675956734 >> v1:172.21.15.5:6789/0 conn(0x7f2ca013f970 legacy=0x7f2ca01420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:02.906 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 10 monclient: _finish_hunting 0 2024-03-29T10:54:02.906 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 1 monclient: found mon.noname-b 2024-03-29T10:54:02.906 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:02.906 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:54:02.906 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 1 -- 172.21.15.5:0/2675956734 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca00aab80 con 0x7f2ca00ae250 2024-03-29T10:54:02.906 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 10 monclient: _finish_auth 0 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 10 monclient: _check_auth_tickets 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca7a89640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca7a89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 1 -- 172.21.15.5:0/2675956734 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2c98001690 con 0x7f2ca00ae250 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.921+0000 7f2ca4823640 10 monclient: dump: 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:02.907 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.908 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:02.908 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:02.908 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:02.908 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:02.908 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:02.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.922+0000 7f2ca7a89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:02.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.923+0000 7f2ca4823640 1 -- 172.21.15.5:0/2675956734 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c98001920 con 0x7f2ca00ae250 2024-03-29T10:54:02.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.923+0000 7f2ca4823640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:02.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.923+0000 7f2ca7a89640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:02.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.923+0000 7f2ca7a89640 4 set_mon_vals no callback set 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.923+0000 7f2ca4823640 1 -- 172.21.15.5:0/2675956734 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2c98001cb0 con 0x7f2ca00ae250 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.923+0000 7f2ca4823640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.923+0000 7f2ca4823640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.923+0000 7f2ca4823640 10 monclient: dump: 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:02.909 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:02.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.924+0000 7f2ca7a89640 10 monclient: shutdown 2024-03-29T10:54:02.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.924+0000 7f2ca7a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:02.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.924+0000 7f2ca7a89640 1 -- 172.21.15.5:0/2675956734 >> v1:172.21.15.5:6790/0 conn(0x7f2ca00ae250 legacy=0x7f2ca013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:02.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.924+0000 7f2ca7a89640 1 -- 172.21.15.5:0/2675956734 shutdown_connections 2024-03-29T10:54:02.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.924+0000 7f2ca7a89640 1 -- 172.21.15.5:0/2675956734 >> 172.21.15.5:0/2675956734 conn(0x7f2ca00b35e0 msgr2=0x7f2ca00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:02.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.924+0000 7f2ca7a89640 1 -- 172.21.15.5:0/2675956734 shutdown_connections 2024-03-29T10:54:02.910 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.924+0000 7f2ca7a89640 1 -- 172.21.15.5:0/2675956734 wait complete. 2024-03-29T10:54:02.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.925+0000 7f2ca7a89640 10 monclient: build_initial_monmap 2024-03-29T10:54:02.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.925+0000 7f2ca7a89640 10 monclient: monmap: 2024-03-29T10:54:02.913 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:02.913 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:02.913 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:54:02.913 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:54:02.913 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:02.913 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:02.913 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:02.914 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:54:02.914 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:54:02.914 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.926+0000 7f2ca7a89640 1 librados: starting msgr at 2024-03-29T10:54:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.926+0000 7f2ca7a89640 1 librados: starting objecter 2024-03-29T10:54:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.926+0000 7f2ca7a89640 1 Processor -- start 2024-03-29T10:54:02.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.926+0000 7f2ca7a89640 1 -- start start 2024-03-29T10:54:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.926+0000 7f2ca7a89640 1 librados: setting wanted keys 2024-03-29T10:54:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.926+0000 7f2ca7a89640 1 librados: calling monclient init 2024-03-29T10:54:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.926+0000 7f2ca7a89640 10 monclient: init 2024-03-29T10:54:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca7a89640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:02.915 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca7a89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca7a89640 10 monclient(hunting): picked mon.noname-c con 0x7f2ca00ae250 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca7a89640 10 monclient(hunting): picked mon.noname-b con 0x7f2ca013f970 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca7a89640 10 monclient(hunting): picked mon.noname-a con 0x7f2ca0142800 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca7a89640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca00ac7a0 con 0x7f2ca0142800 2024-03-29T10:54:02.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca7a89640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca00ac930 con 0x7f2ca013f970 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca7a89640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca0136220 con 0x7f2ca00ae250 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca7a89640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca7a89640 10 monclient(hunting): authenticate will time out at 1256.168288s 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca5825640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f2ca00ae250 0x7f2ca013b4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:49044/0 (socket says 172.21.15.5:49044) 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2ca5825640 1 -- 172.21.15.5:0/3426448289 learned_addr learned my addr 172.21.15.5:0/3426448289 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2c927fc640 10 client.?.objecter ms_handle_connect 0x7f2ca00ae250 2024-03-29T10:54:02.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2c927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968834521 0 0) 0x7f2ca0136220 con 0x7f2ca00ae250 2024-03-29T10:54:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2c927fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2c927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2c927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2c927fc640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:54:02.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.927+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c84004eb0 con 0x7f2ca00ae250 2024-03-29T10:54:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.928+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 619278714 0 0) 0x7f2c84004eb0 con 0x7f2ca00ae250 2024-03-29T10:54:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.928+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ca0136220 con 0x7f2ca00ae250 2024-03-29T10:54:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.928+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2c98001550 con 0x7f2ca00ae250 2024-03-29T10:54:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.928+0000 7f2c927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.928+0000 7f2c927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:02.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.928+0000 7f2c927fc640 10 monclient(hunting): dump: 2024-03-29T10:54:02.919 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:02.919 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:02.920 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:02.920 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:02.920 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.920 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:02.920 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:02.920 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:02.920 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:02.921 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.928+0000 7f2c927fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.928+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 95283416 0 0) 0x7f2ca0136220 con 0x7f2ca00ae250 2024-03-29T10:54:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.928+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 >> v1:172.21.15.177:6789/0 conn(0x7f2ca013f970 legacy=0x7f2ca0135b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:02.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.930+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 >> v1:172.21.15.5:6789/0 conn(0x7f2ca0142800 legacy=0x7f2ca01d7000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:02.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2c927fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:02.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2c927fc640 1 monclient: found mon.c 2024-03-29T10:54:02.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2c927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:02.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2c927fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:02.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca01363b0 con 0x7f2ca00ae250 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2c927fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2c927fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2c927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2ca7a89640 5 monclient: authenticate success, global_id 4505 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2ca7a89640 10 monclient: _renew_subs 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2ca7a89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.932+0000 7f2ca7a89640 1 -- 172.21.15.5:0/3426448289 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca013af30 con 0x7f2ca00ae250 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2ca7a89640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2ca7a89640 10 monclient: _renew_subs 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2ca7a89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2ca7a89640 1 -- 172.21.15.5:0/3426448289 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca01d7710 con 0x7f2ca00ae250 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c980016b0 con 0x7f2ca00ae250 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2ca7a89640 1 librados: init done 2024-03-29T10:54:02.926 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2c927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2c98004e50 con 0x7f2ca00ae250 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2c927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2c927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2c927fc640 10 monclient: dump: 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:02.927 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.933+0000 7f2c93fff640 4 set_mon_vals no callback set 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.935+0000 7f2ca7a89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.935+0000 7f2ca7a89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.935+0000 7f2ca7a89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.935+0000 7f2ca7a89640 1 -- 172.21.15.5:0/3426448289 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c88005080 con 0x7f2ca00ae250 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.936+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f2c9801ae20 con 0x7f2ca00ae250 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.937+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f2c98084c60 con 0x7f2ca00ae250 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.937+0000 7f2c927fc640 10 client.4505.objecter ms_dispatch 0x7f2ca0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.937+0000 7f2c927fc640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.937+0000 7f2c927fc640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.937+0000 7f2c927fc640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.938+0000 7f2ca5024640 10 monclient: get_auth_request con 0x7f2c8406dcc0 auth_method 0 2024-03-29T10:54:02.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.938+0000 7f2c927fc640 10 client.4505.objecter ms_handle_connect 0x7f2c8406dcc0 2024-03-29T10:54:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.945+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c98050540 con 0x7f2ca00ae250 2024-03-29T10:54:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.945+0000 7f2c927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:02.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:02.945+0000 7f2c927fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:03.255 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.274+0000 7f2ca7a89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:03.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.274+0000 7f2ca7a89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:03.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.274+0000 7f2ca7a89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:03.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.274+0000 7f2ca7a89640 1 -- 172.21.15.5:0/3426448289 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c88005330 con 0x7f2ca00ae250 2024-03-29T10:54:03.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.275+0000 7f2c927fc640 1 -- 172.21.15.5:0/3426448289 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f2c98052470 con 0x7f2ca00ae250 2024-03-29T10:54:03.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.275+0000 7f2c927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:03.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.275+0000 7f2c927fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:03.259 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:54:03.259 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:03.262 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.280+0000 7f2ca7a89640 10 librados: watch_flush enter 2024-03-29T10:54:03.262 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.280+0000 7f2ca7a89640 10 librados: watch_flush exit 2024-03-29T10:54:03.262 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.280+0000 7f2ca7a89640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T10:54:03.262 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.280+0000 7f2ca7a89640 10 client.4505.objecter successfully canceled tick 2024-03-29T10:54:03.262 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.281+0000 7f2ca7a89640 1 -- 172.21.15.5:0/3426448289 >> v1:172.21.15.5:6816/2847500151 conn(0x7f2c8406dcc0 legacy=0x7f2c84070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:03.262 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.281+0000 7f2ca7a89640 10 monclient: shutdown 2024-03-29T10:54:03.262 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.281+0000 7f2ca7a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:03.262 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.281+0000 7f2ca7a89640 1 -- 172.21.15.5:0/3426448289 >> v1:172.21.15.5:6790/0 conn(0x7f2ca00ae250 legacy=0x7f2ca013b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:03.263 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.282+0000 7f2ca7a89640 1 -- 172.21.15.5:0/3426448289 shutdown_connections 2024-03-29T10:54:03.263 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.282+0000 7f2ca7a89640 1 -- 172.21.15.5:0/3426448289 >> 172.21.15.5:0/3426448289 conn(0x7f2ca00b35e0 msgr2=0x7f2ca013e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:03.263 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.282+0000 7f2ca7a89640 1 -- 172.21.15.5:0/3426448289 shutdown_connections 2024-03-29T10:54:03.263 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.282+0000 7f2ca7a89640 1 -- 172.21.15.5:0/3426448289 wait complete. 2024-03-29T10:54:03.263 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:03.282+0000 7f2ca7a89640 1 librados: shutdown 2024-03-29T10:54:03.281 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:10.089 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:10.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.259+0000 7f201ffff640 10 monclient: get_monmap_and_config 2024-03-29T10:54:10.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.259+0000 7f201ffff640 10 monclient: build_initial_monmap 2024-03-29T10:54:10.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.260+0000 7f201ffff640 10 monclient: monmap: 2024-03-29T10:54:10.241 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:10.241 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:10.241 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:54:10.261032+0000 2024-03-29T10:54:10.241 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:54:10.261032+0000 2024-03-29T10:54:10.241 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:10.242 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:10.242 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:10.242 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:54:10.242 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:54:10.242 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:10.244 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.262+0000 7f201ffff640 1 Processor -- start 2024-03-29T10:54:10.244 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.262+0000 7f201ffff640 1 -- start start 2024-03-29T10:54:10.244 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.262+0000 7f201ffff640 10 monclient: init 2024-03-29T10:54:10.244 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.263+0000 7f201ffff640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:10.244 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.263+0000 7f201ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:10.244 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.263+0000 7f201ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f20180389e0 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:10.244 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.263+0000 7f201ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f2018038dc0 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:10.244 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.263+0000 7f201ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f2018030660 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:10.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.263+0000 7f201ffff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20180c54c0 con 0x7f20180389e0 2024-03-29T10:54:10.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.263+0000 7f201ffff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2018036430 con 0x7f2018038dc0 2024-03-29T10:54:10.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.263+0000 7f201ffff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2018039840 con 0x7f2018030660 2024-03-29T10:54:10.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.263+0000 7f201ffff640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:10.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.263+0000 7f201ffff640 10 monclient(hunting): authenticate will time out at 1263.504490s 2024-03-29T10:54:10.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.267+0000 7f201effd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f20180389e0 0x7f2018033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44501/0 (socket says 172.21.15.5:44501) 2024-03-29T10:54:10.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.267+0000 7f201effd640 1 -- 172.21.15.5:0/1158141259 learned_addr learned my addr 172.21.15.5:0/1158141259 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:10.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.269+0000 7f201dffb640 1 -- 172.21.15.5:0/1158141259 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211813167 0 0) 0x7f20180c54c0 con 0x7f20180389e0 2024-03-29T10:54:10.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.269+0000 7f201dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:10.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.269+0000 7f201dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:10.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.269+0000 7f201dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:10.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.269+0000 7f201dffb640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:54:10.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.269+0000 7f201dffb640 1 -- 172.21.15.5:0/1158141259 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2008003520 con 0x7f20180389e0 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 1 -- 172.21.15.5:0/1158141259 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1834410234 0 0) 0x7f2008003520 con 0x7f20180389e0 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 1 -- 172.21.15.5:0/1158141259 >> v1:172.21.15.5:6790/0 conn(0x7f2018030660 legacy=0x7f20180d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 1 -- 172.21.15.5:0/1158141259 >> v1:172.21.15.177:6789/0 conn(0x7f2018038dc0 legacy=0x7f20180cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 1 monclient: found mon.noname-a 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 1 -- 172.21.15.5:0/1158141259 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2018035690 con 0x7f20180389e0 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:10.254 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 1 -- 172.21.15.5:0/1158141259 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2014001690 con 0x7f20180389e0 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.272+0000 7f201dffb640 10 monclient: dump: 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:10.255 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:10.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.273+0000 7f201ffff640 5 monclient: authenticate success, global_id 4431 2024-03-29T10:54:10.256 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.273+0000 7f201ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.276+0000 7f201dffb640 1 -- 172.21.15.5:0/1158141259 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20140017f0 con 0x7f20180389e0 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.276+0000 7f201dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.277+0000 7f201dffb640 1 -- 172.21.15.5:0/1158141259 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2014001c40 con 0x7f20180389e0 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.277+0000 7f201dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.277+0000 7f201dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.277+0000 7f201dffb640 10 monclient: dump: 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:10.258 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:10.259 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:10.259 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:10.259 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:10.259 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:10.259 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:10.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.277+0000 7f201ffff640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:10.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.277+0000 7f201ffff640 4 set_mon_vals no callback set 2024-03-29T10:54:10.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.278+0000 7f201ffff640 10 monclient: shutdown 2024-03-29T10:54:10.260 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.278+0000 7f201ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:10.260 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.278+0000 7f201ffff640 1 -- 172.21.15.5:0/1158141259 >> v1:172.21.15.5:6789/0 conn(0x7f20180389e0 legacy=0x7f2018033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:10.260 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.278+0000 7f201ffff640 1 -- 172.21.15.5:0/1158141259 shutdown_connections 2024-03-29T10:54:10.260 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.278+0000 7f201ffff640 1 -- 172.21.15.5:0/1158141259 >> 172.21.15.5:0/1158141259 conn(0x7f2018005c00 msgr2=0x7f2018006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:10.263 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.281+0000 7f201ffff640 1 -- 172.21.15.5:0/1158141259 shutdown_connections 2024-03-29T10:54:10.264 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.283+0000 7f201ffff640 1 -- 172.21.15.5:0/1158141259 wait complete. 2024-03-29T10:54:10.265 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.283+0000 7f201ffff640 10 monclient: build_initial_monmap 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.283+0000 7f201ffff640 10 monclient: monmap: 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:10.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.284+0000 7f201ffff640 1 librados: starting msgr at 2024-03-29T10:54:10.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.284+0000 7f201ffff640 1 librados: starting objecter 2024-03-29T10:54:10.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.284+0000 7f201ffff640 1 Processor -- start 2024-03-29T10:54:10.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.285+0000 7f201ffff640 1 -- start start 2024-03-29T10:54:10.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.285+0000 7f201ffff640 1 librados: setting wanted keys 2024-03-29T10:54:10.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.285+0000 7f201ffff640 1 librados: calling monclient init 2024-03-29T10:54:10.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.285+0000 7f201ffff640 10 monclient: init 2024-03-29T10:54:10.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.285+0000 7f201ffff640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:10.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.285+0000 7f201ffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:10.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.285+0000 7f201ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f2018030660 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:10.272 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.291+0000 7f201ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f20180389e0 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.291+0000 7f201ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f20181619b0 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.291+0000 7f201ffff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20180c3d00 con 0x7f20181619b0 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.291+0000 7f201ffff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2018160e10 con 0x7f20180389e0 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.291+0000 7f201ffff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20180c3960 con 0x7f2018030660 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.291+0000 7f201ffff640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.291+0000 7f201ffff640 10 monclient(hunting): authenticate will time out at 1263.532309s 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.291+0000 7f201e7fc640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f20180389e0 0x7f20180c2b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:49105/0 (socket says 172.21.15.5:49105) 2024-03-29T10:54:10.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.291+0000 7f201e7fc640 1 -- 172.21.15.5:0/2955001531 learned_addr learned my addr 172.21.15.5:0/2955001531 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:10.274 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.293+0000 7f20077fe640 10 client.?.objecter ms_handle_connect 0x7f20180389e0 2024-03-29T10:54:10.274 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.293+0000 7f20077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:10.278 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.295+0000 7f20077fe640 10 client.?.objecter ms_handle_connect 0x7f2018030660 2024-03-29T10:54:10.278 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.295+0000 7f20077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:10.278 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.295+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290241987 0 0) 0x7f2018160e10 con 0x7f20180389e0 2024-03-29T10:54:10.278 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.295+0000 7f20077fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:10.278 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.295+0000 7f20077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:10.278 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.295+0000 7f20077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:10.278 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.295+0000 7f20077fe640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f200c004eb0 con 0x7f20180389e0 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145150674 0 0) 0x7f20180c3960 con 0x7f2018030660 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2018160e10 con 0x7f2018030660 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3037418773 0 0) 0x7f200c004eb0 con 0x7f20180389e0 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20180c3960 con 0x7f20180389e0 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2010001990 con 0x7f20180389e0 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 10 monclient(hunting): dump: 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:10.279 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.296+0000 7f20077fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2073392483 0 0) 0x7f20180c3960 con 0x7f20180389e0 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 >> v1:172.21.15.5:6790/0 conn(0x7f2018030660 legacy=0x7f20181612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:10.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 >> v1:172.21.15.5:6789/0 conn(0x7f20181619b0 legacy=0x7f20180c3250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:10.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:10.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 1 monclient: found mon.b 2024-03-29T10:54:10.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:10.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:54:10.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201800a8a0 con 0x7f20180389e0 2024-03-29T10:54:10.281 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f20077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f201ffff640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f201ffff640 10 monclient: _renew_subs 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f201ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.297+0000 7f201ffff640 1 -- 172.21.15.5:0/2955001531 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201800ab80 con 0x7f20180389e0 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.300+0000 7f201ffff640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.300+0000 7f201ffff640 10 monclient: _renew_subs 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.300+0000 7f201ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.300+0000 7f201ffff640 1 -- 172.21.15.5:0/2955001531 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f201802d150 con 0x7f20180389e0 2024-03-29T10:54:10.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.300+0000 7f201ffff640 1 librados: init done 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.301+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2010001af0 con 0x7f20180389e0 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.301+0000 7f20077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.301+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2010003a60 con 0x7f20180389e0 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.301+0000 7f20077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.301+0000 7f20077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.301+0000 7f201d7fa640 4 set_mon_vals no callback set 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.301+0000 7f20077fe640 10 monclient: dump: 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:10.284 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:10.285 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:10.285 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:10.285 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:10.285 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:10.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.303+0000 7f201ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:10.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.303+0000 7f201ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:10.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.303+0000 7f201ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:54:10.285 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.303+0000 7f201ffff640 1 -- 172.21.15.5:0/2955001531 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2000005080 con 0x7f20180389e0 2024-03-29T10:54:10.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.305+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f2010003c40 con 0x7f20180389e0 2024-03-29T10:54:10.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.306+0000 7f201effd640 10 monclient: get_auth_request con 0x7f200c06dcb0 auth_method 0 2024-03-29T10:54:10.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.306+0000 7f20077fe640 10 client.4402.objecter ms_handle_connect 0x7f200c06dcb0 2024-03-29T10:54:10.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.306+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f2010084ab0 con 0x7f20180389e0 2024-03-29T10:54:10.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.306+0000 7f20077fe640 10 client.4402.objecter ms_dispatch 0x7f2020000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:10.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.306+0000 7f20077fe640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:10.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.306+0000 7f20077fe640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:10.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.307+0000 7f20077fe640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T10:54:10.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.312+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20100501c0 con 0x7f20180389e0 2024-03-29T10:54:10.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.312+0000 7f20077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:10.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.312+0000 7f20077fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:10.676 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.694+0000 7f201ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:10.676 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.694+0000 7f201ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:10.676 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.694+0000 7f201ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:54:10.676 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.694+0000 7f201ffff640 1 -- 172.21.15.5:0/2955001531 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2000005330 con 0x7f20180389e0 2024-03-29T10:54:10.676 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.695+0000 7f20077fe640 1 -- 172.21.15.5:0/2955001531 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f20100522c0 con 0x7f20180389e0 2024-03-29T10:54:10.677 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.695+0000 7f20077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:10.677 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.695+0000 7f20077fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:10.677 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:54:10.677 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:10.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.699+0000 7f201ffff640 10 librados: watch_flush enter 2024-03-29T10:54:10.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.699+0000 7f201ffff640 10 librados: watch_flush exit 2024-03-29T10:54:10.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.699+0000 7f201ffff640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T10:54:10.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.699+0000 7f201ffff640 10 client.4402.objecter successfully canceled tick 2024-03-29T10:54:10.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.700+0000 7f201ffff640 1 -- 172.21.15.5:0/2955001531 >> v1:172.21.15.5:6816/2847500151 conn(0x7f200c06dcb0 legacy=0x7f200c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:10.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.700+0000 7f201ffff640 10 monclient: shutdown 2024-03-29T10:54:10.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.700+0000 7f201ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:10.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.700+0000 7f201ffff640 1 -- 172.21.15.5:0/2955001531 >> v1:172.21.15.177:6789/0 conn(0x7f20180389e0 legacy=0x7f20180c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:10.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.700+0000 7f201ffff640 1 -- 172.21.15.5:0/2955001531 shutdown_connections 2024-03-29T10:54:10.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.700+0000 7f201ffff640 1 -- 172.21.15.5:0/2955001531 >> 172.21.15.5:0/2955001531 conn(0x7f2018005c00 msgr2=0x7f20180c4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:10.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.700+0000 7f201ffff640 1 -- 172.21.15.5:0/2955001531 shutdown_connections 2024-03-29T10:54:10.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.700+0000 7f201ffff640 1 -- 172.21.15.5:0/2955001531 wait complete. 2024-03-29T10:54:10.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:10.701+0000 7f201ffff640 1 librados: shutdown 2024-03-29T10:54:10.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:17.505 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:17.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.678+0000 7f71e8fad640 10 monclient: get_monmap_and_config 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.678+0000 7f71e8fad640 10 monclient: build_initial_monmap 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.678+0000 7f71e8fad640 10 monclient: monmap: 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:54:17.679773+0000 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:54:17.679773+0000 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:54:17.660 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:17.663 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.680+0000 7f71e8fad640 1 Processor -- start 2024-03-29T10:54:17.666 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.683+0000 7f71e8fad640 1 -- start start 2024-03-29T10:54:17.666 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.683+0000 7f71e8fad640 10 monclient: init 2024-03-29T10:54:17.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.684+0000 7f71e8fad640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:17.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.684+0000 7f71e8fad640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:17.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.684+0000 7f71e8fad640 10 monclient(hunting): picked mon.noname-a con 0x7f71dc006a70 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:17.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.684+0000 7f71e8fad640 10 monclient(hunting): picked mon.noname-b con 0x7f71dc02b100 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:17.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.684+0000 7f71e8fad640 10 monclient(hunting): picked mon.noname-c con 0x7f71dc029c30 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:17.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.684+0000 7f71e8fad640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc02a270 con 0x7f71dc006a70 2024-03-29T10:54:17.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.684+0000 7f71e8fad640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc02c440 con 0x7f71dc029c30 2024-03-29T10:54:17.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.684+0000 7f71e8fad640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc0c9440 con 0x7f71dc02b100 2024-03-29T10:54:17.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.684+0000 7f71e8fad640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:17.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.684+0000 7f71e8fad640 10 monclient(hunting): authenticate will time out at 1270.925727s 2024-03-29T10:54:17.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.688+0000 7f71e3fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f71dc029c30 0x7f71dc0d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:56999/0 (socket says 172.21.15.5:56999) 2024-03-29T10:54:17.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.688+0000 7f71e3fff640 1 -- 172.21.15.5:0/91805799 learned_addr learned my addr 172.21.15.5:0/91805799 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:17.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.688+0000 7f71e27fc640 1 -- 172.21.15.5:0/91805799 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212184014 0 0) 0x7f71dc02c440 con 0x7f71dc029c30 2024-03-29T10:54:17.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.688+0000 7f71e27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:17.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.688+0000 7f71e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:17.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.689+0000 7f71e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:17.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.689+0000 7f71e27fc640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:54:17.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.689+0000 7f71e27fc640 1 -- 172.21.15.5:0/91805799 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d4003520 con 0x7f71dc029c30 2024-03-29T10:54:17.671 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.689+0000 7f71e27fc640 1 -- 172.21.15.5:0/91805799 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2183432342 0 0) 0x7f71d4003520 con 0x7f71dc029c30 2024-03-29T10:54:17.671 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.689+0000 7f71e27fc640 1 -- 172.21.15.5:0/91805799 >> v1:172.21.15.5:6790/0 conn(0x7f71dc02b100 legacy=0x7f71dc0cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:17.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.690+0000 7f71e27fc640 1 -- 172.21.15.5:0/91805799 >> v1:172.21.15.5:6789/0 conn(0x7f71dc006a70 legacy=0x7f71dc0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:17.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:17.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 1 monclient: found mon.noname-c 2024-03-29T10:54:17.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:17.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:54:17.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 1 -- 172.21.15.5:0/91805799 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc0c95d0 con 0x7f71dc029c30 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 1 -- 172.21.15.5:0/91805799 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f71e404f730 con 0x7f71dc029c30 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e27fc640 10 monclient: dump: 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:17.673 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:17.674 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:17.674 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:17.674 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:17.674 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e8fad640 5 monclient: authenticate success, global_id 4405 2024-03-29T10:54:17.674 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.691+0000 7f71e8fad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:17.674 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.692+0000 7f71e27fc640 1 -- 172.21.15.5:0/91805799 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71e40008d0 con 0x7f71dc029c30 2024-03-29T10:54:17.675 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.692+0000 7f71e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:17.675 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.692+0000 7f71e8fad640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:17.675 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.692+0000 7f71e8fad640 4 set_mon_vals no callback set 2024-03-29T10:54:17.675 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.693+0000 7f71e8fad640 10 monclient: shutdown 2024-03-29T10:54:17.675 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.693+0000 7f71e8fad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:17.675 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.693+0000 7f71e8fad640 1 -- 172.21.15.5:0/91805799 >> v1:172.21.15.177:6789/0 conn(0x7f71dc029c30 legacy=0x7f71dc0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:17.675 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.693+0000 7f71e8fad640 1 -- 172.21.15.5:0/91805799 shutdown_connections 2024-03-29T10:54:17.675 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.693+0000 7f71e8fad640 1 -- 172.21.15.5:0/91805799 >> 172.21.15.5:0/91805799 conn(0x7f71dc0058f0 msgr2=0x7f71dc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:17.675 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.694+0000 7f71e8fad640 1 -- 172.21.15.5:0/91805799 shutdown_connections 2024-03-29T10:54:17.677 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.695+0000 7f71e8fad640 1 -- 172.21.15.5:0/91805799 wait complete. 2024-03-29T10:54:17.677 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.696+0000 7f71e8fad640 10 monclient: build_initial_monmap 2024-03-29T10:54:17.677 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.696+0000 7f71e8fad640 10 monclient: monmap: 2024-03-29T10:54:17.677 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.696+0000 7f71e8fad640 1 librados: starting msgr at 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.696+0000 7f71e8fad640 1 librados: starting objecter 2024-03-29T10:54:17.678 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.696+0000 7f71e8fad640 1 Processor -- start 2024-03-29T10:54:17.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 1 -- start start 2024-03-29T10:54:17.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 1 librados: setting wanted keys 2024-03-29T10:54:17.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 1 librados: calling monclient init 2024-03-29T10:54:17.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 10 monclient: init 2024-03-29T10:54:17.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:17.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:17.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 10 monclient(hunting): picked mon.noname-a con 0x7f71dc006a70 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:17.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 10 monclient(hunting): picked mon.noname-c con 0x7f71dc029c30 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:17.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 10 monclient(hunting): picked mon.noname-b con 0x7f71dc0ccf90 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:17.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc160b50 con 0x7f71dc006a70 2024-03-29T10:54:17.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc161780 con 0x7f71dc0ccf90 2024-03-29T10:54:17.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc161910 con 0x7f71dc029c30 2024-03-29T10:54:17.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:17.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.697+0000 7f71e8fad640 10 monclient(hunting): authenticate will time out at 1270.938903s 2024-03-29T10:54:17.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.703+0000 7f71e2ffd640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f71dc029c30 0x7f71dc161070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:44771/0 (socket says 172.21.15.5:44771) 2024-03-29T10:54:17.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.703+0000 7f71e2ffd640 1 -- 172.21.15.5:0/2679653077 learned_addr learned my addr 172.21.15.5:0/2679653077 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:17.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.704+0000 7f71cffff640 10 client.?.objecter ms_handle_connect 0x7f71dc029c30 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 client.?.objecter ms_handle_connect 0x7f71dc006a70 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098211592 0 0) 0x7f71dc161910 con 0x7f71dc029c30 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71c8004eb0 con 0x7f71dc029c30 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846438216 0 0) 0x7f71dc160b50 con 0x7f71dc006a70 2024-03-29T10:54:17.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71dc161910 con 0x7f71dc006a70 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3025583542 0 0) 0x7f71c8004eb0 con 0x7f71dc029c30 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71dc160b50 con 0x7f71dc029c30 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f71d0003fb0 con 0x7f71dc029c30 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.705+0000 7f71cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.706+0000 7f71cffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.706+0000 7f71cffff640 10 monclient(hunting): dump: 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:17.689 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.706+0000 7f71cffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.706+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2214525647 0 0) 0x7f71dc161910 con 0x7f71dc006a70 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.706+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71c8004eb0 con 0x7f71dc006a70 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.706+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f71d8001510 con 0x7f71dc006a70 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.706+0000 7f71cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.706+0000 7f71cffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.706+0000 7f71cffff640 10 monclient(hunting): dump: 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:17.690 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:17.691 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:17.691 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:17.691 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:17.691 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:17.691 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:17.691 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:17.691 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:17.691 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:17.691 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.707+0000 7f71cffff640 10 client.?.objecter ms_handle_connect 0x7f71dc0ccf90 2024-03-29T10:54:17.691 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.707+0000 7f71cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543903606 0 0) 0x7f71dc161780 con 0x7f71dc0ccf90 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71dc161910 con 0x7f71dc0ccf90 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3994510676 0 0) 0x7f71c8004eb0 con 0x7f71dc006a70 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 >> v1:172.21.15.5:6790/0 conn(0x7f71dc029c30 legacy=0x7f71dc161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 >> v1:172.21.15.177:6789/0 conn(0x7f71dc0ccf90 legacy=0x7f71dc031eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 1 monclient: found mon.a 2024-03-29T10:54:17.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc0c9980 con 0x7f71dc006a70 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 10 monclient: _finish_auth 0 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.709+0000 7f71cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.710+0000 7f71e8fad640 5 monclient: authenticate success, global_id 4434 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.710+0000 7f71e8fad640 10 monclient: _renew_subs 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.710+0000 7f71e8fad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.710+0000 7f71e8fad640 1 -- 172.21.15.5:0/2679653077 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71dc0325c0 con 0x7f71dc006a70 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.710+0000 7f71e8fad640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.710+0000 7f71e8fad640 10 monclient: _renew_subs 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.710+0000 7f71e8fad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.710+0000 7f71e8fad640 1 -- 172.21.15.5:0/2679653077 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71dc0ca670 con 0x7f71dc006a70 2024-03-29T10:54:17.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.710+0000 7f71e8fad640 1 librados: init done 2024-03-29T10:54:17.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.711+0000 7f71e8fad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:17.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.711+0000 7f71e8fad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:17.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.711+0000 7f71e8fad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:17.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.711+0000 7f71e8fad640 1 -- 172.21.15.5:0/2679653077 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71dc0c9b70 con 0x7f71dc006a70 2024-03-29T10:54:17.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.712+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71d8001a20 con 0x7f71dc006a70 2024-03-29T10:54:17.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.712+0000 7f71cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.712+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f71d8003910 con 0x7f71dc006a70 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.712+0000 7f71cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.712+0000 7f71cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.712+0000 7f71cffff640 10 monclient: dump: 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:17.695 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:17.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.712+0000 7f71e1ffb640 4 set_mon_vals no callback set 2024-03-29T10:54:17.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.715+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f71d80198e0 con 0x7f71dc006a70 2024-03-29T10:54:17.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.716+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f71d8083740 con 0x7f71dc006a70 2024-03-29T10:54:17.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.716+0000 7f71cffff640 10 client.4434.objecter ms_dispatch 0x7f71e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:17.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.716+0000 7f71cffff640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:17.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.716+0000 7f71cffff640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:17.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.716+0000 7f71cffff640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T10:54:17.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.718+0000 7f71e2ffd640 10 monclient: get_auth_request con 0x7f71c806dd70 auth_method 0 2024-03-29T10:54:17.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.720+0000 7f71cffff640 10 client.4434.objecter ms_handle_connect 0x7f71c806dd70 2024-03-29T10:54:17.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.721+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71d804f020 con 0x7f71dc006a70 2024-03-29T10:54:17.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.721+0000 7f71cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:17.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:17.721+0000 7f71cffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:18.072 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.091+0000 7f71e8fad640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:18.072 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.091+0000 7f71e8fad640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:18.072 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.091+0000 7f71e8fad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:18.072 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.091+0000 7f71e8fad640 1 -- 172.21.15.5:0/2679653077 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71dc1609b0 con 0x7f71dc006a70 2024-03-29T10:54:18.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.093+0000 7f71cffff640 1 -- 172.21.15.5:0/2679653077 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f71dc1609b0 con 0x7f71dc006a70 2024-03-29T10:54:18.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.093+0000 7f71cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:18.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.093+0000 7f71cffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:18.075 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:54:18.075 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:18.080 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.099+0000 7f71e8fad640 10 librados: watch_flush enter 2024-03-29T10:54:18.080 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.099+0000 7f71e8fad640 10 librados: watch_flush exit 2024-03-29T10:54:18.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.099+0000 7f71e8fad640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T10:54:18.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.099+0000 7f71e8fad640 10 client.4434.objecter successfully canceled tick 2024-03-29T10:54:18.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.101+0000 7f71e8fad640 1 -- 172.21.15.5:0/2679653077 >> v1:172.21.15.5:6816/2847500151 conn(0x7f71c806dd70 legacy=0x7f71c8070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:18.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.101+0000 7f71e8fad640 10 monclient: shutdown 2024-03-29T10:54:18.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.101+0000 7f71e8fad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:18.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.101+0000 7f71e8fad640 1 -- 172.21.15.5:0/2679653077 >> v1:172.21.15.5:6789/0 conn(0x7f71dc006a70 legacy=0x7f71dc0cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:18.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.101+0000 7f71e8fad640 1 -- 172.21.15.5:0/2679653077 shutdown_connections 2024-03-29T10:54:18.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.101+0000 7f71e8fad640 1 -- 172.21.15.5:0/2679653077 >> 172.21.15.5:0/2679653077 conn(0x7f71dc0058f0 msgr2=0x7f71dc0050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:18.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.102+0000 7f71e8fad640 1 -- 172.21.15.5:0/2679653077 shutdown_connections 2024-03-29T10:54:18.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.102+0000 7f71e8fad640 1 -- 172.21.15.5:0/2679653077 wait complete. 2024-03-29T10:54:18.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:18.102+0000 7f71e8fad640 1 librados: shutdown 2024-03-29T10:54:18.100 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:24.906 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.077+0000 7fbd0d525640 10 monclient: get_monmap_and_config 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.077+0000 7fbd0d525640 10 monclient: build_initial_monmap 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.077+0000 7fbd0d525640 10 monclient: monmap: 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:54:25.078936+0000 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:54:25.078936+0000 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:25.059 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:54:25.060 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:54:25.060 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:25.063 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.079+0000 7fbd0d525640 1 Processor -- start 2024-03-29T10:54:25.063 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.079+0000 7fbd0d525640 1 -- start start 2024-03-29T10:54:25.063 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.079+0000 7fbd0d525640 10 monclient: init 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.079+0000 7fbd0d525640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.079+0000 7fbd0d525640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.079+0000 7fbd0d525640 10 monclient(hunting): picked mon.noname-c con 0x7fbd080b1f60 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.079+0000 7fbd0d525640 10 monclient(hunting): picked mon.noname-a con 0x7fbd08071eb0 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.080+0000 7fbd0d525640 10 monclient(hunting): picked mon.noname-b con 0x7fbd08143d10 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.080+0000 7fbd0d525640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0806ef80 con 0x7fbd08071eb0 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.080+0000 7fbd0d525640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0806f260 con 0x7fbd080b1f60 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.080+0000 7fbd0d525640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd080b40c0 con 0x7fbd08143d10 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.080+0000 7fbd0d525640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.080+0000 7fbd0d525640 10 monclient(hunting): authenticate will time out at 1278.321122s 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.080+0000 7fbd077fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fbd08143d10 0x7fbd08146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:38046/0 (socket says 172.21.15.5:38046) 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.080+0000 7fbd077fe640 1 -- 172.21.15.5:0/3107785307 learned_addr learned my addr 172.21.15.5:0/3107785307 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.081+0000 7fbd05ffb640 1 -- 172.21.15.5:0/3107785307 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239864568 0 0) 0x7fbd080b40c0 con 0x7fbd08143d10 2024-03-29T10:54:25.064 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.081+0000 7fbd05ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.065 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.081+0000 7fbd05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.065 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.081+0000 7fbd05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.065 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.081+0000 7fbd05ffb640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:54:25.065 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.081+0000 7fbd05ffb640 1 -- 172.21.15.5:0/3107785307 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf0003520 con 0x7fbd08143d10 2024-03-29T10:54:25.065 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.081+0000 7fbd05ffb640 1 -- 172.21.15.5:0/3107785307 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4080548089 0 0) 0x7fbcf0003520 con 0x7fbd08143d10 2024-03-29T10:54:25.065 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.081+0000 7fbd05ffb640 1 -- 172.21.15.5:0/3107785307 >> v1:172.21.15.177:6789/0 conn(0x7fbd080b1f60 legacy=0x7fbd08143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 1 -- 172.21.15.5:0/3107785307 >> v1:172.21.15.5:6789/0 conn(0x7fbd08071eb0 legacy=0x7fbd08146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 1 monclient: found mon.noname-b 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 1 -- 172.21.15.5:0/3107785307 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd08155d70 con 0x7fbd08143d10 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 1 -- 172.21.15.5:0/3107785307 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbd00001a10 con 0x7fbd08143d10 2024-03-29T10:54:25.066 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd05ffb640 10 monclient: dump: 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd0d525640 5 monclient: authenticate success, global_id 4514 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.083+0000 7fbd0d525640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:25.067 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.084+0000 7fbd05ffb640 1 -- 172.21.15.5:0/3107785307 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd00001be0 con 0x7fbd08143d10 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.084+0000 7fbd05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.084+0000 7fbd0d525640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.084+0000 7fbd0d525640 4 set_mon_vals no callback set 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.084+0000 7fbd05ffb640 1 -- 172.21.15.5:0/3107785307 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbd00002030 con 0x7fbd08143d10 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.084+0000 7fbd05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.084+0000 7fbd05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.084+0000 7fbd05ffb640 10 monclient: dump: 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:25.068 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:25.069 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:25.069 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:25.069 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:25.069 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.085+0000 7fbd0d525640 10 monclient: shutdown 2024-03-29T10:54:25.069 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.085+0000 7fbd0d525640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:25.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.085+0000 7fbd0d525640 1 -- 172.21.15.5:0/3107785307 >> v1:172.21.15.5:6790/0 conn(0x7fbd08143d10 legacy=0x7fbd08146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.085+0000 7fbd0d525640 1 -- 172.21.15.5:0/3107785307 shutdown_connections 2024-03-29T10:54:25.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.085+0000 7fbd0d525640 1 -- 172.21.15.5:0/3107785307 >> 172.21.15.5:0/3107785307 conn(0x7fbd08066a00 msgr2=0x7fbd08066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:25.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.085+0000 7fbd0d525640 1 -- 172.21.15.5:0/3107785307 shutdown_connections 2024-03-29T10:54:25.070 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.089+0000 7fbd0d525640 1 -- 172.21.15.5:0/3107785307 wait complete. 2024-03-29T10:54:25.072 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.090+0000 7fbd0d525640 10 monclient: build_initial_monmap 2024-03-29T10:54:25.076 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.090+0000 7fbd0d525640 10 monclient: monmap: 2024-03-29T10:54:25.076 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.090+0000 7fbd0d525640 1 librados: starting msgr at 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.090+0000 7fbd0d525640 1 librados: starting objecter 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 1 Processor -- start 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 1 -- start start 2024-03-29T10:54:25.077 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 1 librados: setting wanted keys 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 1 librados: calling monclient init 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 10 monclient: init 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 10 monclient(hunting): picked mon.noname-a con 0x7fbd080b1f60 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 10 monclient(hunting): picked mon.noname-b con 0x7fbd08143d10 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 10 monclient(hunting): picked mon.noname-c con 0x7fbd081547f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd080acb70 con 0x7fbd080b1f60 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0808b230 con 0x7fbd08143d10 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd080acda0 con 0x7fbd081547f0 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.091+0000 7fbd0d525640 10 monclient(hunting): authenticate will time out at 1278.333041s 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.092+0000 7fbd077fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fbd081547f0 0x7fbd0808ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:43940/0 (socket says 172.21.15.5:43940) 2024-03-29T10:54:25.078 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.092+0000 7fbd077fe640 1 -- 172.21.15.5:0/1894549796 learned_addr learned my addr 172.21.15.5:0/1894549796 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.092+0000 7fbcef7fe640 10 client.?.objecter ms_handle_connect 0x7fbd081547f0 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.092+0000 7fbcef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.092+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983036716 0 0) 0x7fbd080acda0 con 0x7fbd081547f0 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.092+0000 7fbcef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.092+0000 7fbcef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.092+0000 7fbcef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.092+0000 7fbcef7fe640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.092+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf4004eb0 con 0x7fbd081547f0 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.093+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2051966680 0 0) 0x7fbcf4004eb0 con 0x7fbd081547f0 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.093+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd080acda0 con 0x7fbd081547f0 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.093+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbd00001ef0 con 0x7fbd081547f0 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.093+0000 7fbcef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.093+0000 7fbcef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:25.079 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.093+0000 7fbcef7fe640 10 monclient(hunting): dump: 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.093+0000 7fbcef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.093+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3233190557 0 0) 0x7fbd080acda0 con 0x7fbd081547f0 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 >> v1:172.21.15.177:6789/0 conn(0x7fbd08143d10 legacy=0x7fbd0808a410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 >> v1:172.21.15.5:6789/0 conn(0x7fbd080b1f60 legacy=0x7fbd080ad510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:25.080 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 1 monclient: found mon.c 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd080ad080 con 0x7fbd081547f0 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbd0d525640 5 monclient: authenticate success, global_id 4517 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbd0d525640 10 monclient: _renew_subs 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbd0d525640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbd0d525640 1 -- 172.21.15.5:0/1894549796 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd080adc20 con 0x7fbd081547f0 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbd0d525640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbd0d525640 10 monclient: _renew_subs 2024-03-29T10:54:25.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbd0d525640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbd0d525640 1 -- 172.21.15.5:0/1894549796 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd080abd80 con 0x7fbd081547f0 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbd0d525640 1 librados: init done 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd00015eb0 con 0x7fbd081547f0 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbcef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.094+0000 7fbd057fa640 4 set_mon_vals no callback set 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.095+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbd00016280 con 0x7fbd081547f0 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.095+0000 7fbcef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.095+0000 7fbcef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.095+0000 7fbcef7fe640 10 monclient: dump: 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:25.082 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.083 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:25.083 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:25.083 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:25.083 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:25.083 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:25.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.096+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fbd0002c250 con 0x7fbd081547f0 2024-03-29T10:54:25.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.097+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fbd00095f00 con 0x7fbd081547f0 2024-03-29T10:54:25.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.097+0000 7fbcef7fe640 10 client.4517.objecter ms_dispatch 0x7fbd08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:25.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.097+0000 7fbcef7fe640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:25.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.097+0000 7fbcef7fe640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:25.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.097+0000 7fbcef7fe640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-29T10:54:25.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.097+0000 7fbd0d525640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:25.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.097+0000 7fbd0d525640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:25.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.097+0000 7fbd0d525640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:25.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.097+0000 7fbd0d525640 1 -- 172.21.15.5:0/1894549796 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce8005080 con 0x7fbd081547f0 2024-03-29T10:54:25.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.099+0000 7fbd06ffd640 10 monclient: get_auth_request con 0x7fbcf406dc10 auth_method 0 2024-03-29T10:54:25.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.099+0000 7fbcef7fe640 10 client.4517.objecter ms_handle_connect 0x7fbcf406dc10 2024-03-29T10:54:25.088 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.107+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd000617e0 con 0x7fbd081547f0 2024-03-29T10:54:25.088 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.107+0000 7fbcef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:25.088 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.107+0000 7fbcef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:25.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.492+0000 7fbd0d525640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:25.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.492+0000 7fbd0d525640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:25.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.492+0000 7fbd0d525640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:25.475 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.492+0000 7fbd0d525640 1 -- 172.21.15.5:0/1894549796 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbce8005330 con 0x7fbd081547f0 2024-03-29T10:54:25.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.495+0000 7fbcef7fe640 1 -- 172.21.15.5:0/1894549796 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fbd00063710 con 0x7fbd081547f0 2024-03-29T10:54:25.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.495+0000 7fbcef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:25.476 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.495+0000 7fbcef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:25.476 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:54:25.477 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:25.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.499+0000 7fbd0d525640 10 librados: watch_flush enter 2024-03-29T10:54:25.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.499+0000 7fbd0d525640 10 librados: watch_flush exit 2024-03-29T10:54:25.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.499+0000 7fbd0d525640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-29T10:54:25.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.499+0000 7fbd0d525640 10 client.4517.objecter successfully canceled tick 2024-03-29T10:54:25.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.500+0000 7fbd0d525640 1 -- 172.21.15.5:0/1894549796 >> v1:172.21.15.5:6816/2847500151 conn(0x7fbcf406dc10 legacy=0x7fbcf40700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.500+0000 7fbd0d525640 10 monclient: shutdown 2024-03-29T10:54:25.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.500+0000 7fbd0d525640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:25.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.500+0000 7fbd0d525640 1 -- 172.21.15.5:0/1894549796 >> v1:172.21.15.5:6790/0 conn(0x7fbd081547f0 legacy=0x7fbd0808ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.503+0000 7fbd0d525640 1 -- 172.21.15.5:0/1894549796 shutdown_connections 2024-03-29T10:54:25.484 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.503+0000 7fbd0d525640 1 -- 172.21.15.5:0/1894549796 >> 172.21.15.5:0/1894549796 conn(0x7fbd08066a00 msgr2=0x7fbd08066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:25.489 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.507+0000 7fbd0d525640 1 -- 172.21.15.5:0/1894549796 shutdown_connections 2024-03-29T10:54:25.491 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.509+0000 7fbd0d525640 1 -- 172.21.15.5:0/1894549796 wait complete. 2024-03-29T10:54:25.491 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:25.510+0000 7fbd0d525640 1 librados: shutdown 2024-03-29T10:54:25.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:32.319 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:32.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.482+0000 7fc043fff640 10 monclient: get_monmap_and_config 2024-03-29T10:54:32.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.482+0000 7fc043fff640 10 monclient: build_initial_monmap 2024-03-29T10:54:32.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.483+0000 7fc043fff640 10 monclient: monmap: 2024-03-29T10:54:32.464 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:32.464 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:32.464 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:54:32.483985+0000 2024-03-29T10:54:32.464 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:54:32.483985+0000 2024-03-29T10:54:32.464 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:32.464 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:32.464 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:32.465 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:54:32.465 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:54:32.465 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:32.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.484+0000 7fc043fff640 1 Processor -- start 2024-03-29T10:54:32.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.484+0000 7fc043fff640 1 -- start start 2024-03-29T10:54:32.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.484+0000 7fc043fff640 10 monclient: init 2024-03-29T10:54:32.466 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc043fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc043fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc043fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc0440b2800 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc043fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc0440a9c00 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc043fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc0440b38c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc043fff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc044141eb0 con 0x7fc0440a9c00 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc043fff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0440b22d0 con 0x7fc0440b2800 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc043fff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc044142d30 con 0x7fc0440b38c0 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc043fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc043fff640 10 monclient(hunting): authenticate will time out at 1285.726557s 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc042ffd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fc0440b2800 0x7fc044142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:59472/0 (socket says 172.21.15.5:59472) 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc0437fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fc0440b38c0 0x7fc04414d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:48004/0 (socket says 172.21.15.5:48004) 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.485+0000 7fc042ffd640 1 -- 172.21.15.5:0/605552475 learned_addr learned my addr 172.21.15.5:0/605552475 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 1 -- 172.21.15.5:0/605552475 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844403757 0 0) 0x7fc0440b22d0 con 0x7fc0440b2800 2024-03-29T10:54:32.467 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:32.468 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:32.468 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:32.468 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:54:32.468 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 1 -- 172.21.15.5:0/605552475 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc02c003520 con 0x7fc0440b2800 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 1 -- 172.21.15.5:0/605552475 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 545960926 0 0) 0x7fc02c003520 con 0x7fc0440b2800 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 1 -- 172.21.15.5:0/605552475 >> v1:172.21.15.5:6790/0 conn(0x7fc0440b38c0 legacy=0x7fc04414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 1 -- 172.21.15.5:0/605552475 >> v1:172.21.15.5:6789/0 conn(0x7fc0440a9c00 legacy=0x7fc04413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 1 monclient: found mon.noname-c 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 1 -- 172.21.15.5:0/605552475 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440aaf90 con 0x7fc0440b2800 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc043fff640 5 monclient: authenticate success, global_id 4411 2024-03-29T10:54:32.469 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc043fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 1 -- 172.21.15.5:0/605552475 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc03c001690 con 0x7fc0440b2800 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient: dump: 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:32.470 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc043fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 1 -- 172.21.15.5:0/605552475 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc03c0018b0 con 0x7fc0440b2800 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc041ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc043fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.486+0000 7fc043fff640 4 set_mon_vals no callback set 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc041ffb640 1 -- 172.21.15.5:0/605552475 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc03c001bc0 con 0x7fc0440b2800 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc041ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc041ffb640 10 monclient: dump: 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:32.471 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:32.472 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:32.472 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:32.472 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:32.472 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:32.472 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc043fff640 10 monclient: shutdown 2024-03-29T10:54:32.472 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc043fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:32.472 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc043fff640 1 -- 172.21.15.5:0/605552475 >> v1:172.21.15.177:6789/0 conn(0x7fc0440b2800 legacy=0x7fc044142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.472 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc043fff640 1 -- 172.21.15.5:0/605552475 shutdown_connections 2024-03-29T10:54:32.472 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc043fff640 1 -- 172.21.15.5:0/605552475 >> 172.21.15.5:0/605552475 conn(0x7fc044066a00 msgr2=0x7fc044066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:32.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc043fff640 1 -- 172.21.15.5:0/605552475 shutdown_connections 2024-03-29T10:54:32.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.487+0000 7fc043fff640 1 -- 172.21.15.5:0/605552475 wait complete. 2024-03-29T10:54:32.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.488+0000 7fc043fff640 10 monclient: build_initial_monmap 2024-03-29T10:54:32.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.488+0000 7fc043fff640 10 monclient: monmap: 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.489+0000 7fc043fff640 1 librados: starting msgr at 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.489+0000 7fc043fff640 1 librados: starting objecter 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.489+0000 7fc043fff640 1 Processor -- start 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.489+0000 7fc043fff640 1 -- start start 2024-03-29T10:54:32.479 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.489+0000 7fc043fff640 1 librados: setting wanted keys 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.489+0000 7fc043fff640 1 librados: calling monclient init 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.489+0000 7fc043fff640 10 monclient: init 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc043fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc043fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc043fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc0440a9c00 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc043fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc0440b2800 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc043fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc0440b38c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc043fff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04413aec0 con 0x7fc0440b38c0 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc0427fc640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fc0440b2800 0x7fc04413a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:48156/0 (socket says 172.21.15.5:48156) 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc0427fc640 1 -- 172.21.15.5:0/37040328 learned_addr learned my addr 172.21.15.5:0/37040328 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc044072000 con 0x7fc0440a9c00 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc02b7fe640 10 client.?.objecter ms_handle_connect 0x7fc0440b2800 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc02b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:32.480 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc044070f10 con 0x7fc0440b2800 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc043fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.490+0000 7fc043fff640 10 monclient(hunting): authenticate will time out at 1285.732022s 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 10 client.?.objecter ms_handle_connect 0x7fc0440b38c0 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496728404 0 0) 0x7fc044070f10 con 0x7fc0440b2800 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc030004eb0 con 0x7fc0440b2800 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854395256 0 0) 0x7fc04413aec0 con 0x7fc0440b38c0 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:32.481 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 10 monclient(hunting): my global_id is 4440 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc044070f10 con 0x7fc0440b38c0 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.491+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1665600586 0 0) 0x7fc030004eb0 con 0x7fc0440b2800 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc04413aec0 con 0x7fc0440b2800 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc038001950 con 0x7fc0440b2800 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 10 monclient(hunting): dump: 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:32.482 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 10 client.?.objecter ms_handle_connect 0x7fc0440a9c00 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 74855529 0 0) 0x7fc04413aec0 con 0x7fc0440b2800 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 >> v1:172.21.15.177:6789/0 conn(0x7fc0440a9c00 legacy=0x7fc0441db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 >> v1:172.21.15.5:6789/0 conn(0x7fc0440b38c0 legacy=0x7fc04413a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 1 monclient: found mon.c 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440704b0 con 0x7fc0440b2800 2024-03-29T10:54:32.483 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:32.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:32.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.492+0000 7fc02b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:32.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc043fff640 5 monclient: authenticate success, global_id 4523 2024-03-29T10:54:32.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc043fff640 10 monclient: _renew_subs 2024-03-29T10:54:32.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc043fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:32.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc044070760 con 0x7fc0440b2800 2024-03-29T10:54:32.485 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc043fff640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc043fff640 10 monclient: _renew_subs 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc043fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc044070040 con 0x7fc0440b2800 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc043fff640 1 librados: init done 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc038003610 con 0x7fc0440b2800 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc02b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc0380039a0 con 0x7fc0440b2800 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc02b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc02b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc02b7fe640 10 monclient: dump: 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:32.486 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.493+0000 7fc0417fa640 4 set_mon_vals no callback set 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.494+0000 7fc043fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.495+0000 7fc043fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.495+0000 7fc043fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.495+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc024005080 con 0x7fc0440b2800 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.495+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fc038003b00 con 0x7fc0440b2800 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.496+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fc038084a50 con 0x7fc0440b2800 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.496+0000 7fc02b7fe640 10 client.4523.objecter ms_dispatch 0x7fc044000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:32.487 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.496+0000 7fc02b7fe640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:32.488 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.496+0000 7fc02b7fe640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:32.488 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.496+0000 7fc02b7fe640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T10:54:32.488 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.496+0000 7fc042ffd640 10 monclient: get_auth_request con 0x7fc03006dce0 auth_method 0 2024-03-29T10:54:32.488 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.497+0000 7fc02b7fe640 10 client.4523.objecter ms_handle_connect 0x7fc03006dce0 2024-03-29T10:54:32.488 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.506+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc038050330 con 0x7fc0440b2800 2024-03-29T10:54:32.488 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.506+0000 7fc02b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:32.488 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.506+0000 7fc02b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:32.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.891+0000 7fc043fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:32.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.891+0000 7fc043fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:32.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.891+0000 7fc043fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:54:32.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.891+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc024005330 con 0x7fc0440b2800 2024-03-29T10:54:32.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.893+0000 7fc02b7fe640 1 -- 172.21.15.5:0/37040328 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fc038052260 con 0x7fc0440b2800 2024-03-29T10:54:32.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.893+0000 7fc02b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:32.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.893+0000 7fc02b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:32.874 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:54:32.875 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:32.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.896+0000 7fc043fff640 10 librados: watch_flush enter 2024-03-29T10:54:32.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.896+0000 7fc043fff640 10 librados: watch_flush exit 2024-03-29T10:54:32.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.896+0000 7fc043fff640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T10:54:32.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.896+0000 7fc043fff640 10 client.4523.objecter successfully canceled tick 2024-03-29T10:54:32.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.896+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 >> v1:172.21.15.5:6816/2847500151 conn(0x7fc03006dce0 legacy=0x7fc0300701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.896+0000 7fc043fff640 10 monclient: shutdown 2024-03-29T10:54:32.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.896+0000 7fc043fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:32.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.896+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 >> v1:172.21.15.5:6790/0 conn(0x7fc0440b2800 legacy=0x7fc04413a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.897+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 shutdown_connections 2024-03-29T10:54:32.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.897+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 >> 172.21.15.5:0/37040328 conn(0x7fc044066a00 msgr2=0x7fc044140590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:32.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.897+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 shutdown_connections 2024-03-29T10:54:32.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.897+0000 7fc043fff640 1 -- 172.21.15.5:0/37040328 wait complete. 2024-03-29T10:54:32.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:32.897+0000 7fc043fff640 1 librados: shutdown 2024-03-29T10:54:32.895 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:39.702 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:39.838 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.857+0000 7f893320e640 10 monclient: get_monmap_and_config 2024-03-29T10:54:39.838 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.857+0000 7f893320e640 10 monclient: build_initial_monmap 2024-03-29T10:54:39.838 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.857+0000 7f893320e640 10 monclient: monmap: 2024-03-29T10:54:39.838 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:39.838 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:39.838 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:54:39.858129+0000 2024-03-29T10:54:39.838 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:54:39.858129+0000 2024-03-29T10:54:39.838 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:39.838 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:39.839 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:39.839 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:54:39.839 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:54:39.839 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:39.839 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.857+0000 7f893320e640 1 Processor -- start 2024-03-29T10:54:39.839 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.858+0000 7f893320e640 1 -- start start 2024-03-29T10:54:39.839 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.858+0000 7f893320e640 10 monclient: init 2024-03-29T10:54:39.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.859+0000 7f893320e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:39.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.859+0000 7f893320e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:39.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.859+0000 7f893320e640 10 monclient(hunting): picked mon.noname-c con 0x7f892c06ade0 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:39.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.859+0000 7f893320e640 10 monclient(hunting): picked mon.noname-a con 0x7f892c0afa90 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:39.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.860+0000 7f893320e640 10 monclient(hunting): picked mon.noname-b con 0x7f892c0ae550 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:39.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.860+0000 7f893320e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0b0b20 con 0x7f892c0afa90 2024-03-29T10:54:39.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.860+0000 7f893320e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c13d8c0 con 0x7f892c06ade0 2024-03-29T10:54:39.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.860+0000 7f893320e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c13da50 con 0x7f892c0ae550 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.860+0000 7f893320e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.860+0000 7f893320e640 10 monclient(hunting): authenticate will time out at 1293.101567s 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.861+0000 7f892bfff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f892c0afa90 0x7f892c13d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58564/0 (socket says 172.21.15.5:58564) 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.861+0000 7f892bfff640 1 -- 172.21.15.5:0/1584510217 learned_addr learned my addr 172.21.15.5:0/1584510217 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.861+0000 7f892b7fe640 1 -- 172.21.15.5:0/1584510217 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082809900 0 0) 0x7f892c0b0b20 con 0x7f892c0afa90 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.861+0000 7f892b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.861+0000 7f892b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.861+0000 7f892b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.861+0000 7f892b7fe640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.861+0000 7f892b7fe640 1 -- 172.21.15.5:0/1584510217 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f891c003520 con 0x7f892c0afa90 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.862+0000 7f892b7fe640 1 -- 172.21.15.5:0/1584510217 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3134411579 0 0) 0x7f891c003520 con 0x7f892c0afa90 2024-03-29T10:54:39.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.862+0000 7f892b7fe640 1 -- 172.21.15.5:0/1584510217 >> v1:172.21.15.5:6790/0 conn(0x7f892c0ae550 legacy=0x7f892c14b730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.862+0000 7f892b7fe640 1 -- 172.21.15.5:0/1584510217 >> v1:172.21.15.177:6789/0 conn(0x7f892c06ade0 legacy=0x7f892c143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 1 monclient: found mon.noname-a 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 1 -- 172.21.15.5:0/1584510217 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c072570 con 0x7f892c0afa90 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 1 -- 172.21.15.5:0/1584510217 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8918001670 con 0x7f892c0afa90 2024-03-29T10:54:39.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f892b7fe640 10 monclient: dump: 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f893320e640 5 monclient: authenticate success, global_id 4443 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.863+0000 7f893320e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:39.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.864+0000 7f892b7fe640 1 -- 172.21.15.5:0/1584510217 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89180018d0 con 0x7f892c0afa90 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.864+0000 7f892b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.864+0000 7f892b7fe640 1 -- 172.21.15.5:0/1584510217 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8918001d20 con 0x7f892c0afa90 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.864+0000 7f892b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.864+0000 7f892b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.864+0000 7f892b7fe640 10 monclient: dump: 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:39.849 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:39.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.864+0000 7f893320e640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:39.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.864+0000 7f893320e640 4 set_mon_vals no callback set 2024-03-29T10:54:39.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.865+0000 7f893320e640 10 monclient: shutdown 2024-03-29T10:54:39.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.865+0000 7f893320e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:39.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.865+0000 7f893320e640 1 -- 172.21.15.5:0/1584510217 >> v1:172.21.15.5:6789/0 conn(0x7f892c0afa90 legacy=0x7f892c13d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.865+0000 7f893320e640 1 -- 172.21.15.5:0/1584510217 shutdown_connections 2024-03-29T10:54:39.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.865+0000 7f893320e640 1 -- 172.21.15.5:0/1584510217 >> 172.21.15.5:0/1584510217 conn(0x7f892c0b4930 msgr2=0x7f892c0b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:39.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.865+0000 7f893320e640 1 -- 172.21.15.5:0/1584510217 shutdown_connections 2024-03-29T10:54:39.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.865+0000 7f893320e640 1 -- 172.21.15.5:0/1584510217 wait complete. 2024-03-29T10:54:39.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.866+0000 7f893320e640 10 monclient: build_initial_monmap 2024-03-29T10:54:39.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.866+0000 7f893320e640 10 monclient: monmap: 2024-03-29T10:54:39.852 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:39.852 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:39.852 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.867+0000 7f893320e640 1 librados: starting msgr at 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.867+0000 7f893320e640 1 librados: starting objecter 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.867+0000 7f893320e640 1 Processor -- start 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.868+0000 7f893320e640 1 -- start start 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.868+0000 7f893320e640 1 librados: setting wanted keys 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.868+0000 7f893320e640 1 librados: calling monclient init 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.868+0000 7f893320e640 10 monclient: init 2024-03-29T10:54:39.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f893320e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f893320e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f893320e640 10 monclient(hunting): picked mon.noname-c con 0x7f892c06ade0 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f893320e640 10 monclient(hunting): picked mon.noname-a con 0x7f892c0ae550 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f893320e640 10 monclient(hunting): picked mon.noname-b con 0x7f892c0afa90 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f893320e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0b4670 con 0x7f892c0ae550 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f893320e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c134930 con 0x7f892c0afa90 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f893320e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c135800 con 0x7f892c06ade0 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f893320e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f893320e640 10 monclient(hunting): authenticate will time out at 1293.110567s 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f89317ab640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f892c0afa90 0x7f892c1e2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:52684/0 (socket says 172.21.15.5:52684) 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.869+0000 7f89317ab640 1 -- 172.21.15.5:0/1655261186 learned_addr learned my addr 172.21.15.5:0/1655261186 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.870+0000 7f89297fa640 10 client.?.objecter ms_handle_connect 0x7f892c0afa90 2024-03-29T10:54:39.854 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.870+0000 7f89297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:39.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.871+0000 7f89297fa640 10 client.?.objecter ms_handle_connect 0x7f892c0ae550 2024-03-29T10:54:39.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.872+0000 7f89297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:39.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.872+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900353842 0 0) 0x7f892c0b4670 con 0x7f892c0ae550 2024-03-29T10:54:39.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.872+0000 7f89297fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:39.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.872+0000 7f89297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.872+0000 7f89297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.872+0000 7f89297fa640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.872+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8910004eb0 con 0x7f892c0ae550 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212869255 0 0) 0x7f892c134930 con 0x7f892c0afa90 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c0b4670 con 0x7f892c0afa90 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2692564918 0 0) 0x7f8910004eb0 con 0x7f892c0ae550 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f892c134930 con 0x7f892c0ae550 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8918001d20 con 0x7f892c0ae550 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:39.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 10 monclient(hunting): dump: 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1978782480 0 0) 0x7f892c134930 con 0x7f892c0ae550 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 >> v1:172.21.15.5:6790/0 conn(0x7f892c06ade0 legacy=0x7f892c1350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.873+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 >> v1:172.21.15.177:6789/0 conn(0x7f892c0afa90 legacy=0x7f892c1e2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:39.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f89297fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f89297fa640 1 monclient: found mon.a 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f89297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f89297fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c1372d0 con 0x7f892c0ae550 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f89297fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f89297fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f89297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f893320e640 5 monclient: authenticate success, global_id 4446 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f893320e640 10 monclient: _renew_subs 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f893320e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f893320e640 1 -- 172.21.15.5:0/1655261186 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c1374e0 con 0x7f892c0ae550 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f893320e640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f893320e640 10 monclient: _renew_subs 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f893320e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:39.860 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f893320e640 1 -- 172.21.15.5:0/1655261186 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c0b4390 con 0x7f892c0ae550 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f893320e640 1 librados: init done 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8918004d20 con 0x7f892c0ae550 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f89297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.874+0000 7f892a7fc640 4 set_mon_vals no callback set 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.875+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8918005030 con 0x7f892c0ae550 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.875+0000 7f89297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.875+0000 7f89297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.875+0000 7f89297fa640 10 monclient: dump: 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:39.861 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:39.862 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:39.862 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:39.862 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:39.862 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.875+0000 7f8916ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:39.862 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.875+0000 7f8916ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:39.862 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.875+0000 7f8916ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:39.862 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.875+0000 7f8916ffd640 1 -- 172.21.15.5:0/1655261186 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c075f10 con 0x7f892c0ae550 2024-03-29T10:54:39.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.877+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f891801b000 con 0x7f892c0ae550 2024-03-29T10:54:39.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.878+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f8918084bf0 con 0x7f892c0ae550 2024-03-29T10:54:39.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.878+0000 7f89297fa640 10 client.4446.objecter ms_dispatch 0x7f892c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:39.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.878+0000 7f89297fa640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:39.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.878+0000 7f89297fa640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:39.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.878+0000 7f89297fa640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-29T10:54:39.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.878+0000 7f8930faa640 10 monclient: get_auth_request con 0x7f891006daa0 auth_method 0 2024-03-29T10:54:39.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.879+0000 7f89297fa640 10 client.4446.objecter ms_handle_connect 0x7f891006daa0 2024-03-29T10:54:39.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.885+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89180504d0 con 0x7f892c0ae550 2024-03-29T10:54:39.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.885+0000 7f89297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:39.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:39.885+0000 7f89297fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:40.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.214+0000 7f8916ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:40.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.214+0000 7f8916ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:40.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.214+0000 7f8916ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:40.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.214+0000 7f8916ffd640 1 -- 172.21.15.5:0/1655261186 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f892c13a650 con 0x7f892c0ae550 2024-03-29T10:54:40.197 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:54:40.197 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:40.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.215+0000 7f89297fa640 1 -- 172.21.15.5:0/1655261186 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f8918052400 con 0x7f892c0ae550 2024-03-29T10:54:40.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.215+0000 7f89297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:40.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.215+0000 7f89297fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:40.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.218+0000 7f8916ffd640 10 librados: watch_flush enter 2024-03-29T10:54:40.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.218+0000 7f8916ffd640 10 librados: watch_flush exit 2024-03-29T10:54:40.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.218+0000 7f8916ffd640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-29T10:54:40.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.218+0000 7f8916ffd640 10 client.4446.objecter successfully canceled tick 2024-03-29T10:54:40.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.218+0000 7f8916ffd640 1 -- 172.21.15.5:0/1655261186 >> v1:172.21.15.5:6816/2847500151 conn(0x7f891006daa0 legacy=0x7f891006ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.218+0000 7f8916ffd640 10 monclient: shutdown 2024-03-29T10:54:40.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.218+0000 7f8916ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:40.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.219+0000 7f8916ffd640 1 -- 172.21.15.5:0/1655261186 >> v1:172.21.15.5:6789/0 conn(0x7f892c0ae550 legacy=0x7f892c133ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.220+0000 7f8916ffd640 1 -- 172.21.15.5:0/1655261186 shutdown_connections 2024-03-29T10:54:40.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.220+0000 7f8916ffd640 1 -- 172.21.15.5:0/1655261186 >> 172.21.15.5:0/1655261186 conn(0x7f892c0b4930 msgr2=0x7f892c0b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:40.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.220+0000 7f8916ffd640 1 -- 172.21.15.5:0/1655261186 shutdown_connections 2024-03-29T10:54:40.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.220+0000 7f8916ffd640 1 -- 172.21.15.5:0/1655261186 wait complete. 2024-03-29T10:54:40.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:40.220+0000 7f8916ffd640 1 librados: shutdown 2024-03-29T10:54:40.218 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:47.026 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:47.175 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.193+0000 7f2edf658640 10 monclient: get_monmap_and_config 2024-03-29T10:54:47.175 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.193+0000 7f2edf658640 10 monclient: build_initial_monmap 2024-03-29T10:54:47.175 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.193+0000 7f2edf658640 10 monclient: monmap: 2024-03-29T10:54:47.175 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:47.175 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:47.175 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:54:47.194831+0000 2024-03-29T10:54:47.175 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:54:47.194831+0000 2024-03-29T10:54:47.175 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:47.175 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:47.175 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:47.176 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:54:47.176 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:54:47.176 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:47.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.194+0000 7f2edf658640 1 Processor -- start 2024-03-29T10:54:47.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.194+0000 7f2edf658640 1 -- start start 2024-03-29T10:54:47.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.195+0000 7f2edf658640 10 monclient: init 2024-03-29T10:54:47.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.195+0000 7f2edf658640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:47.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.195+0000 7f2edf658640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:47.177 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.195+0000 7f2edf658640 10 monclient(hunting): picked mon.noname-b con 0x7f2ed81457f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:47.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.196+0000 7f2edf658640 10 monclient(hunting): picked mon.noname-a con 0x7f2ed813f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:47.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.196+0000 7f2edf658640 10 monclient(hunting): picked mon.noname-c con 0x7f2ed80a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:47.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.196+0000 7f2edf658640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed80aaef0 con 0x7f2ed813f650 2024-03-29T10:54:47.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.196+0000 7f2edf658640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed8145bd0 con 0x7f2ed80a9d70 2024-03-29T10:54:47.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.196+0000 7f2edf658640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed8145eb0 con 0x7f2ed81457f0 2024-03-29T10:54:47.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.196+0000 7f2edf658640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:47.178 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.196+0000 7f2edf658640 10 monclient(hunting): authenticate will time out at 1300.437514s 2024-03-29T10:54:47.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.199+0000 7f2edd3f4640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f2ed81457f0 0x7f2ed813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:56209/0 (socket says 172.21.15.5:56209) 2024-03-29T10:54:47.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.199+0000 7f2edd3f4640 1 -- 172.21.15.5:0/4206102934 learned_addr learned my addr 172.21.15.5:0/4206102934 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.199+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695041483 0 0) 0x7f2ed80aaef0 con 0x7f2ed813f650 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec4003520 con 0x7f2ed813f650 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695044741 0 0) 0x7f2ed8145eb0 con 0x7f2ed81457f0 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:54:47.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed80aaef0 con 0x7f2ed81457f0 2024-03-29T10:54:47.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2518564765 0 0) 0x7f2ec4003520 con 0x7f2ed813f650 2024-03-29T10:54:47.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 >> v1:172.21.15.5:6790/0 conn(0x7f2ed81457f0 legacy=0x7f2ed813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 >> v1:172.21.15.177:6789/0 conn(0x7f2ed80a9d70 legacy=0x7f2ed814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:47.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 1 monclient: found mon.noname-a 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed8146340 con 0x7f2ed813f650 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient: _finish_auth 0 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2ed4001690 con 0x7f2ed813f650 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:47.184 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.200+0000 7f2ecffff640 10 monclient: dump: 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.201+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ed40017f0 con 0x7f2ed813f650 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.201+0000 7f2ecffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.201+0000 7f2ecffff640 1 -- 172.21.15.5:0/4206102934 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2ed4001c00 con 0x7f2ed813f650 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.201+0000 7f2ecffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.201+0000 7f2ecffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.201+0000 7f2ecffff640 10 monclient: dump: 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.202+0000 7f2edf658640 5 monclient: authenticate success, global_id 4449 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.202+0000 7f2edf658640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:47.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.202+0000 7f2edf658640 4 set_mon_vals no callback set 2024-03-29T10:54:47.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.203+0000 7f2edf658640 10 monclient: shutdown 2024-03-29T10:54:47.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.203+0000 7f2edf658640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:47.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.203+0000 7f2edf658640 1 -- 172.21.15.5:0/4206102934 >> v1:172.21.15.5:6789/0 conn(0x7f2ed813f650 legacy=0x7f2ed80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.206+0000 7f2edf658640 1 -- 172.21.15.5:0/4206102934 shutdown_connections 2024-03-29T10:54:47.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.206+0000 7f2edf658640 1 -- 172.21.15.5:0/4206102934 >> 172.21.15.5:0/4206102934 conn(0x7f2ed8066a00 msgr2=0x7f2ed8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:47.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.206+0000 7f2edf658640 1 -- 172.21.15.5:0/4206102934 shutdown_connections 2024-03-29T10:54:47.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.206+0000 7f2edf658640 1 -- 172.21.15.5:0/4206102934 wait complete. 2024-03-29T10:54:47.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.208+0000 7f2edf658640 10 monclient: build_initial_monmap 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.208+0000 7f2edf658640 10 monclient: monmap: 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:54:47.189 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:47.190 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.208+0000 7f2edf658640 1 librados: starting msgr at 2024-03-29T10:54:47.190 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.208+0000 7f2edf658640 1 librados: starting objecter 2024-03-29T10:54:47.190 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.209+0000 7f2edf658640 1 Processor -- start 2024-03-29T10:54:47.190 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.209+0000 7f2edf658640 1 -- start start 2024-03-29T10:54:47.190 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.209+0000 7f2edf658640 1 librados: setting wanted keys 2024-03-29T10:54:47.191 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.209+0000 7f2edf658640 1 librados: calling monclient init 2024-03-29T10:54:47.191 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.209+0000 7f2edf658640 10 monclient: init 2024-03-29T10:54:47.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.210+0000 7f2edf658640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:47.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.210+0000 7f2edf658640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:47.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.210+0000 7f2edf658640 10 monclient(hunting): picked mon.noname-a con 0x7f2ed80a9d70 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:47.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.210+0000 7f2edf658640 10 monclient(hunting): picked mon.noname-b con 0x7f2ed813f650 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:47.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.210+0000 7f2edf658640 10 monclient(hunting): picked mon.noname-c con 0x7f2ed81457f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:47.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.210+0000 7f2edf658640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed81d3530 con 0x7f2ed80a9d70 2024-03-29T10:54:47.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.210+0000 7f2edf658640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed81d3780 con 0x7f2ed813f650 2024-03-29T10:54:47.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.210+0000 7f2edf658640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed81d43b0 con 0x7f2ed81457f0 2024-03-29T10:54:47.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.210+0000 7f2edf658640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:47.192 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.210+0000 7f2edf658640 10 monclient(hunting): authenticate will time out at 1300.452016s 2024-03-29T10:54:47.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.211+0000 7f2edcbf3640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f2ed813f650 0x7f2ed8136dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:42986/0 (socket says 172.21.15.5:42986) 2024-03-29T10:54:47.193 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.211+0000 7f2edcbf3640 1 -- 172.21.15.5:0/3803544206 learned_addr learned my addr 172.21.15.5:0/3803544206 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:47.196 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.214+0000 7f2ecdffb640 10 client.?.objecter ms_handle_connect 0x7f2ed813f650 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.214+0000 7f2ecdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.214+0000 7f2ecdffb640 10 client.?.objecter ms_handle_connect 0x7f2ed80a9d70 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.214+0000 7f2ecdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.214+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042490133 0 0) 0x7f2ed81d3530 con 0x7f2ed80a9d70 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.214+0000 7f2ecdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.214+0000 7f2ecdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.214+0000 7f2ecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.214+0000 7f2ecdffb640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.215+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ebc004eb0 con 0x7f2ed80a9d70 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.215+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120846395 0 0) 0x7f2ed81d3780 con 0x7f2ed813f650 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.215+0000 7f2ecdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.215+0000 7f2ecdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.215+0000 7f2ecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.215+0000 7f2ecdffb640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:54:47.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.215+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ebc0051d0 con 0x7f2ed813f650 2024-03-29T10:54:47.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 19314379 0 0) 0x7f2ebc004eb0 con 0x7f2ed80a9d70 2024-03-29T10:54:47.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ed81d3780 con 0x7f2ed80a9d70 2024-03-29T10:54:47.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2ed0001c10 con 0x7f2ed80a9d70 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 10 monclient(hunting): dump: 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:47.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 10 client.?.objecter ms_handle_connect 0x7f2ed81457f0 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2723188154 0 0) 0x7f2ebc0051d0 con 0x7f2ed813f650 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ebc004eb0 con 0x7f2ed813f650 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2ed4001c00 con 0x7f2ed813f650 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 10 monclient(hunting): dump: 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.202 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 435023500 0 0) 0x7f2ed81d3780 con 0x7f2ed80a9d70 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 >> v1:172.21.15.5:6790/0 conn(0x7f2ed81457f0 legacy=0x7f2ed81ded70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.216+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 >> v1:172.21.15.177:6789/0 conn(0x7f2ed813f650 legacy=0x7f2ed8136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 1 monclient: found mon.a 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed81420a0 con 0x7f2ed80a9d70 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:47.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2edf658640 5 monclient: authenticate success, global_id 4452 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2edf658640 10 monclient: _renew_subs 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2edf658640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2edf658640 1 -- 172.21.15.5:0/3803544206 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed81374d0 con 0x7f2ed80a9d70 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2edf658640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2edf658640 10 monclient: _renew_subs 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2edf658640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2edf658640 1 -- 172.21.15.5:0/3803544206 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed806e2e0 con 0x7f2ed80a9d70 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2edf658640 1 librados: init done 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ed00020e0 con 0x7f2ed80a9d70 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2ed0004d00 con 0x7f2ed80a9d70 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:47.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecdffb640 10 monclient: dump: 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:47.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.217+0000 7f2ecf7fe640 4 set_mon_vals no callback set 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.220+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f2ed001acd0 con 0x7f2ed80a9d70 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.221+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f2ed00549d0 con 0x7f2ed80a9d70 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.221+0000 7f2edf658640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.221+0000 7f2edf658640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.221+0000 7f2edf658640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.221+0000 7f2edf658640 1 -- 172.21.15.5:0/3803544206 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb8005080 con 0x7f2ed80a9d70 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.221+0000 7f2ecdffb640 10 client.4452.objecter ms_dispatch 0x7f2ed8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.222+0000 7f2ecdffb640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.222+0000 7f2ecdffb640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.222+0000 7f2edcbf3640 10 monclient: get_auth_request con 0x7f2ebc06e020 auth_method 0 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.222+0000 7f2ecdffb640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-29T10:54:47.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.222+0000 7f2ecdffb640 10 client.4452.objecter ms_handle_connect 0x7f2ebc06e020 2024-03-29T10:54:47.211 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.230+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ed0050c70 con 0x7f2ed80a9d70 2024-03-29T10:54:47.211 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.230+0000 7f2ecdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:47.211 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.230+0000 7f2ecdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:47.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.551+0000 7f2edf658640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:47.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.551+0000 7f2edf658640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:47.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.551+0000 7f2edf658640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:47.532 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.551+0000 7f2edf658640 1 -- 172.21.15.5:0/3803544206 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eb8005330 con 0x7f2ed80a9d70 2024-03-29T10:54:47.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.551+0000 7f2ecdffb640 1 -- 172.21.15.5:0/3803544206 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f2ed0050c70 con 0x7f2ed80a9d70 2024-03-29T10:54:47.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.551+0000 7f2ecdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:47.533 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.551+0000 7f2ecdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:47.533 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:54:47.533 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:47.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.554+0000 7f2edf658640 10 librados: watch_flush enter 2024-03-29T10:54:47.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.554+0000 7f2edf658640 10 librados: watch_flush exit 2024-03-29T10:54:47.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.554+0000 7f2edf658640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-29T10:54:47.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.554+0000 7f2edf658640 10 client.4452.objecter successfully canceled tick 2024-03-29T10:54:47.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.554+0000 7f2edf658640 1 -- 172.21.15.5:0/3803544206 >> v1:172.21.15.5:6816/2847500151 conn(0x7f2ebc06e020 legacy=0x7f2ebc0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.554+0000 7f2edf658640 10 monclient: shutdown 2024-03-29T10:54:47.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.554+0000 7f2edf658640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:47.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.554+0000 7f2edf658640 1 -- 172.21.15.5:0/3803544206 >> v1:172.21.15.5:6789/0 conn(0x7f2ed80a9d70 legacy=0x7f2ed81d3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.554+0000 7f2edf658640 1 -- 172.21.15.5:0/3803544206 shutdown_connections 2024-03-29T10:54:47.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.555+0000 7f2edf658640 1 -- 172.21.15.5:0/3803544206 >> 172.21.15.5:0/3803544206 conn(0x7f2ed8066a00 msgr2=0x7f2ed813b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:47.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.555+0000 7f2edf658640 1 -- 172.21.15.5:0/3803544206 shutdown_connections 2024-03-29T10:54:47.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.555+0000 7f2edf658640 1 -- 172.21.15.5:0/3803544206 wait complete. 2024-03-29T10:54:47.537 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:47.555+0000 7f2edf658640 1 librados: shutdown 2024-03-29T10:54:47.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:54.358 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:54.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.527+0000 7f4b349e2640 10 monclient: get_monmap_and_config 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.527+0000 7f4b349e2640 10 monclient: build_initial_monmap 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.527+0000 7f4b349e2640 10 monclient: monmap: 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:54:54.528321+0000 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:54:54.528321+0000 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:54:54.511 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.528+0000 7f4b349e2640 1 Processor -- start 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.528+0000 7f4b349e2640 1 -- start start 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.528+0000 7f4b349e2640 10 monclient: init 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.529+0000 7f4b349e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.529+0000 7f4b349e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.529+0000 7f4b349e2640 10 monclient(hunting): picked mon.noname-c con 0x7f4b300ad6c0 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.529+0000 7f4b349e2640 10 monclient(hunting): picked mon.noname-a con 0x7f4b300af6c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.529+0000 7f4b349e2640 10 monclient(hunting): picked mon.noname-b con 0x7f4b3014fdc0 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.529+0000 7f4b349e2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b300a94e0 con 0x7f4b300af6c0 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.529+0000 7f4b349e2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b300af3a0 con 0x7f4b300ad6c0 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.529+0000 7f4b349e2640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b300afb60 con 0x7f4b3014fdc0 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.529+0000 7f4b349e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:54.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.529+0000 7f4b349e2640 10 monclient(hunting): authenticate will time out at 1307.770511s 2024-03-29T10:54:54.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.530+0000 7f4b2ed9d640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f4b3014fdc0 0x7f4b3015a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:45406/0 (socket says 172.21.15.5:45406) 2024-03-29T10:54:54.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.530+0000 7f4b2ed9d640 1 -- 172.21.15.5:0/4170283274 learned_addr learned my addr 172.21.15.5:0/4170283274 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:54.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.531+0000 7f4b2d59a640 1 -- 172.21.15.5:0/4170283274 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870484303 0 0) 0x7f4b300afb60 con 0x7f4b3014fdc0 2024-03-29T10:54:54.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.531+0000 7f4b2d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:54:54.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.531+0000 7f4b2d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:54.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.531+0000 7f4b2d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:54.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.531+0000 7f4b2d59a640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:54:54.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.531+0000 7f4b2d59a640 1 -- 172.21.15.5:0/4170283274 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b18003520 con 0x7f4b3014fdc0 2024-03-29T10:54:54.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 1 -- 172.21.15.5:0/4170283274 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1958923149 0 0) 0x7f4b18003520 con 0x7f4b3014fdc0 2024-03-29T10:54:54.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 1 -- 172.21.15.5:0/4170283274 >> v1:172.21.15.177:6789/0 conn(0x7f4b300ad6c0 legacy=0x7f4b3014f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:54.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 1 -- 172.21.15.5:0/4170283274 >> v1:172.21.15.5:6789/0 conn(0x7f4b300af6c0 legacy=0x7f4b30152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:54.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:54:54.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 1 monclient: found mon.noname-b 2024-03-29T10:54:54.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:54.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:54:54.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 1 -- 172.21.15.5:0/4170283274 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b300aab80 con 0x7f4b3014fdc0 2024-03-29T10:54:54.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 10 monclient: _finish_auth 0 2024-03-29T10:54:54.517 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 1 -- 172.21.15.5:0/4170283274 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4b200012f0 con 0x7f4b3014fdc0 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.533+0000 7f4b2d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.534+0000 7f4b2d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.534+0000 7f4b2d59a640 10 monclient: dump: 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:54.518 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.534+0000 7f4b349e2640 5 monclient: authenticate success, global_id 4532 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.534+0000 7f4b349e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.534+0000 7f4b2d59a640 1 -- 172.21.15.5:0/4170283274 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b20001580 con 0x7f4b3014fdc0 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.534+0000 7f4b2d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.534+0000 7f4b349e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.534+0000 7f4b349e2640 4 set_mon_vals no callback set 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.535+0000 7f4b349e2640 10 monclient: shutdown 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.535+0000 7f4b349e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.535+0000 7f4b349e2640 1 -- 172.21.15.5:0/4170283274 >> v1:172.21.15.5:6790/0 conn(0x7f4b3014fdc0 legacy=0x7f4b3015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.535+0000 7f4b349e2640 1 -- 172.21.15.5:0/4170283274 shutdown_connections 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.535+0000 7f4b349e2640 1 -- 172.21.15.5:0/4170283274 >> 172.21.15.5:0/4170283274 conn(0x7f4b300b35e0 msgr2=0x7f4b300b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.535+0000 7f4b349e2640 1 -- 172.21.15.5:0/4170283274 shutdown_connections 2024-03-29T10:54:54.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.535+0000 7f4b349e2640 1 -- 172.21.15.5:0/4170283274 wait complete. 2024-03-29T10:54:54.521 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.537+0000 7f4b349e2640 10 monclient: build_initial_monmap 2024-03-29T10:54:54.521 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.537+0000 7f4b349e2640 10 monclient: monmap: 2024-03-29T10:54:54.521 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:54:54.521 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:54.521 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:54:54.521 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:54:54.521 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:54.521 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:54.521 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.537+0000 7f4b349e2640 1 librados: starting msgr at 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.537+0000 7f4b349e2640 1 librados: starting objecter 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.537+0000 7f4b349e2640 1 Processor -- start 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.537+0000 7f4b349e2640 1 -- start start 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.537+0000 7f4b349e2640 1 librados: setting wanted keys 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.537+0000 7f4b349e2640 1 librados: calling monclient init 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.537+0000 7f4b349e2640 10 monclient: init 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b349e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b349e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b349e2640 10 monclient(hunting): picked mon.noname-a con 0x7f4b300ad6c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:54:54.522 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b349e2640 10 monclient(hunting): picked mon.noname-b con 0x7f4b300af6c0 addr v1:172.21.15.177:6789/0 2024-03-29T10:54:54.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b349e2640 10 monclient(hunting): picked mon.noname-c con 0x7f4b3014fdc0 addr v1:172.21.15.5:6790/0 2024-03-29T10:54:54.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b349e2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b30072050 con 0x7f4b300ad6c0 2024-03-29T10:54:54.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b349e2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b301ec2a0 con 0x7f4b300af6c0 2024-03-29T10:54:54.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b349e2640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b301e8de0 con 0x7f4b3014fdc0 2024-03-29T10:54:54.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b349e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:54.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b349e2640 10 monclient(hunting): authenticate will time out at 1307.780396s 2024-03-29T10:54:54.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b2ed9d640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f4b3014fdc0 0x7f4b301f3f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:51360/0 (socket says 172.21.15.5:51360) 2024-03-29T10:54:54.523 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.539+0000 7f4b2ed9d640 1 -- 172.21.15.5:0/1253420693 learned_addr learned my addr 172.21.15.5:0/1253420693 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:54:54.524 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.543+0000 7f4b1effd640 10 client.?.objecter ms_handle_connect 0x7f4b3014fdc0 2024-03-29T10:54:54.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.544+0000 7f4b1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:54.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.544+0000 7f4b1effd640 10 client.?.objecter ms_handle_connect 0x7f4b300ad6c0 2024-03-29T10:54:54.525 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.544+0000 7f4b1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:54.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.544+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399392244 0 0) 0x7f4b301e8de0 con 0x7f4b3014fdc0 2024-03-29T10:54:54.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:54.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:54.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:54.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:54:54.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b10004eb0 con 0x7f4b3014fdc0 2024-03-29T10:54:54.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374004401 0 0) 0x7f4b30072050 con 0x7f4b300ad6c0 2024-03-29T10:54:54.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:54.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:54.526 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:54.527 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:54:54.527 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b301e8de0 con 0x7f4b300ad6c0 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 181782785 0 0) 0x7f4b301e8de0 con 0x7f4b300ad6c0 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.545+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b30072050 con 0x7f4b300ad6c0 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.546+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4b24001870 con 0x7f4b300ad6c0 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.546+0000 7f4b1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.546+0000 7f4b1effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.546+0000 7f4b1effd640 10 monclient(hunting): dump: 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:54.529 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.546+0000 7f4b1effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.546+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3062385062 0 0) 0x7f4b10004eb0 con 0x7f4b3014fdc0 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.546+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b301e8de0 con 0x7f4b3014fdc0 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.546+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3186707526 0 0) 0x7f4b30072050 con 0x7f4b300ad6c0 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.546+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 >> v1:172.21.15.5:6790/0 conn(0x7f4b3014fdc0 legacy=0x7f4b301f3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 >> v1:172.21.15.177:6789/0 conn(0x7f4b300af6c0 legacy=0x7f4b3008a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b1effd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b1effd640 1 monclient: found mon.a 2024-03-29T10:54:54.530 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b1effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b30072470 con 0x7f4b300ad6c0 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b1effd640 10 monclient: _finish_auth 0 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b1effd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b349e2640 5 monclient: authenticate success, global_id 4458 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b349e2640 10 monclient: _renew_subs 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b349e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b349e2640 1 -- 172.21.15.5:0/1253420693 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b301ec7c0 con 0x7f4b300ad6c0 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b349e2640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b349e2640 10 monclient: _renew_subs 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b349e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b349e2640 1 -- 172.21.15.5:0/1253420693 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3014c0e0 con 0x7f4b300ad6c0 2024-03-29T10:54:54.531 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.547+0000 7f4b349e2640 1 librados: init done 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.548+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b240035f0 con 0x7f4b300ad6c0 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.548+0000 7f4b1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.548+0000 7f4b2cd99640 4 set_mon_vals no callback set 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.548+0000 7f4b349e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.550+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4b24001d80 con 0x7f4b300ad6c0 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.550+0000 7f4b1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.550+0000 7f4b1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.551+0000 7f4b1effd640 10 monclient: dump: 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:54:54.534 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.551+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f4b240197b0 con 0x7f4b300ad6c0 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.551+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f4b240837f0 con 0x7f4b300ad6c0 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.552+0000 7f4b1effd640 10 client.4458.objecter ms_dispatch 0x7f4b30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.552+0000 7f4b349e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.552+0000 7f4b349e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.552+0000 7f4b349e2640 1 -- 172.21.15.5:0/1253420693 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b14005080 con 0x7f4b300ad6c0 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.552+0000 7f4b1effd640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:54.535 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.552+0000 7f4b1effd640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:54.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.552+0000 7f4b1effd640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-29T10:54:54.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.552+0000 7f4b2dd9b640 10 monclient: get_auth_request con 0x7f4b1006ddc0 auth_method 0 2024-03-29T10:54:54.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.552+0000 7f4b1effd640 10 client.4458.objecter ms_handle_connect 0x7f4b1006ddc0 2024-03-29T10:54:54.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.560+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b2404f150 con 0x7f4b300ad6c0 2024-03-29T10:54:54.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.560+0000 7f4b1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:54.541 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.560+0000 7f4b1effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.893+0000 7f4b349e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.893+0000 7f4b349e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.893+0000 7f4b349e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:54:54.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.893+0000 7f4b349e2640 1 -- 172.21.15.5:0/1253420693 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b14005330 con 0x7f4b300ad6c0 2024-03-29T10:54:54.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.894+0000 7f4b1effd640 1 -- 172.21.15.5:0/1253420693 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f4b24051080 con 0x7f4b300ad6c0 2024-03-29T10:54:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.894+0000 7f4b1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:54.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.894+0000 7f4b1effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:54.877 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:54:54.878 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:54.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.899+0000 7f4b349e2640 10 librados: watch_flush enter 2024-03-29T10:54:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.900+0000 7f4b349e2640 10 librados: watch_flush exit 2024-03-29T10:54:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.900+0000 7f4b349e2640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-29T10:54:54.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.900+0000 7f4b349e2640 10 client.4458.objecter successfully canceled tick 2024-03-29T10:54:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.900+0000 7f4b349e2640 1 -- 172.21.15.5:0/1253420693 >> v1:172.21.15.5:6816/2847500151 conn(0x7f4b1006ddc0 legacy=0x7f4b10070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.900+0000 7f4b349e2640 10 monclient: shutdown 2024-03-29T10:54:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.900+0000 7f4b349e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.900+0000 7f4b349e2640 1 -- 172.21.15.5:0/1253420693 >> v1:172.21.15.5:6789/0 conn(0x7f4b300ad6c0 legacy=0x7f4b301e86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.900+0000 7f4b349e2640 1 -- 172.21.15.5:0/1253420693 shutdown_connections 2024-03-29T10:54:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.901+0000 7f4b349e2640 1 -- 172.21.15.5:0/1253420693 >> 172.21.15.5:0/1253420693 conn(0x7f4b300b35e0 msgr2=0x7f4b3014f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.901+0000 7f4b349e2640 1 -- 172.21.15.5:0/1253420693 shutdown_connections 2024-03-29T10:54:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.901+0000 7f4b349e2640 1 -- 172.21.15.5:0/1253420693 wait complete. 2024-03-29T10:54:54.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:54:54.901+0000 7f4b349e2640 1 librados: shutdown 2024-03-29T10:54:54.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:01.706 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:01.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.872+0000 7f30f7a84640 10 monclient: get_monmap_and_config 2024-03-29T10:55:01.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.872+0000 7f30f7a84640 10 monclient: build_initial_monmap 2024-03-29T10:55:01.853 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.872+0000 7f30f7a84640 10 monclient: monmap: 2024-03-29T10:55:01.853 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:01.853 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:01.853 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:55:01.873071+0000 2024-03-29T10:55:01.853 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:55:01.873071+0000 2024-03-29T10:55:01.854 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:01.854 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:01.854 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:01.854 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:55:01.854 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:55:01.854 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:01.856 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.876+0000 7f30f7a84640 1 Processor -- start 2024-03-29T10:55:01.857 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.876+0000 7f30f7a84640 1 -- start start 2024-03-29T10:55:01.858 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.876+0000 7f30f7a84640 10 monclient: init 2024-03-29T10:55:01.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.877+0000 7f30f7a84640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:01.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.877+0000 7f30f7a84640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:01.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.877+0000 7f30f7a84640 10 monclient(hunting): picked mon.noname-b con 0x7f30e8036ac0 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:01.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.877+0000 7f30f7a84640 10 monclient(hunting): picked mon.noname-a con 0x7f30e8035310 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:01.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.878+0000 7f30f7a84640 10 monclient(hunting): picked mon.noname-c con 0x7f30e802cd20 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:01.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.878+0000 7f30f7a84640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e802d400 con 0x7f30e8035310 2024-03-29T10:55:01.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.878+0000 7f30f7a84640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e802e3a0 con 0x7f30e802cd20 2024-03-29T10:55:01.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.878+0000 7f30f7a84640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e80c30f0 con 0x7f30e8036ac0 2024-03-29T10:55:01.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.878+0000 7f30f7a84640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:01.859 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.878+0000 7f30f7a84640 10 monclient(hunting): authenticate will time out at 1315.119122s 2024-03-29T10:55:01.862 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.881+0000 7f30f5820640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f30e8036ac0 0x7f30e80c29e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:40998/0 (socket says 172.21.15.5:40998) 2024-03-29T10:55:01.862 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.881+0000 7f30f5820640 1 -- 172.21.15.5:0/3482405105 learned_addr learned my addr 172.21.15.5:0/3482405105 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:01.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.882+0000 7f30f481e640 1 -- 172.21.15.5:0/3482405105 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642456971 0 0) 0x7f30e80c30f0 con 0x7f30e8036ac0 2024-03-29T10:55:01.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.882+0000 7f30f481e640 10 monclient(hunting): _negotiate 2024-03-29T10:55:01.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.882+0000 7f30f481e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:01.863 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.882+0000 7f30f481e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:01.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.882+0000 7f30f481e640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:55:01.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.882+0000 7f30f481e640 1 -- 172.21.15.5:0/3482405105 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30dc003520 con 0x7f30e8036ac0 2024-03-29T10:55:01.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.882+0000 7f30f481e640 1 -- 172.21.15.5:0/3482405105 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1284280201 0 0) 0x7f30dc003520 con 0x7f30e8036ac0 2024-03-29T10:55:01.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.882+0000 7f30f481e640 1 -- 172.21.15.5:0/3482405105 >> v1:172.21.15.177:6789/0 conn(0x7f30e802cd20 legacy=0x7f30e80c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:01.864 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.882+0000 7f30f481e640 1 -- 172.21.15.5:0/3482405105 >> v1:172.21.15.5:6789/0 conn(0x7f30e8035310 legacy=0x7f30e80c57c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:01.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 10 monclient: _finish_hunting 0 2024-03-29T10:55:01.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 1 monclient: found mon.noname-b 2024-03-29T10:55:01.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:01.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:55:01.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 1 -- 172.21.15.5:0/3482405105 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e80c3280 con 0x7f30e8036ac0 2024-03-29T10:55:01.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 10 monclient: _finish_auth 0 2024-03-29T10:55:01.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 10 monclient: _check_auth_tickets 2024-03-29T10:55:01.865 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 1 -- 172.21.15.5:0/3482405105 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f30ec001690 con 0x7f30e8036ac0 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f7a84640 5 monclient: authenticate success, global_id 4538 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f7a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 10 monclient: dump: 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.866 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:01.867 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:01.867 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:01.867 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:01.867 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:01.867 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f7a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:01.867 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 1 -- 172.21.15.5:0/3482405105 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30ec0017f0 con 0x7f30e8036ac0 2024-03-29T10:55:01.867 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f481e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:01.867 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f7a84640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:01.867 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.883+0000 7f30f7a84640 4 set_mon_vals no callback set 2024-03-29T10:55:01.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.884+0000 7f30f7a84640 10 monclient: shutdown 2024-03-29T10:55:01.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.884+0000 7f30f7a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:01.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.884+0000 7f30f7a84640 1 -- 172.21.15.5:0/3482405105 >> v1:172.21.15.5:6790/0 conn(0x7f30e8036ac0 legacy=0x7f30e80c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.885+0000 7f30f7a84640 1 -- 172.21.15.5:0/3482405105 shutdown_connections 2024-03-29T10:55:01.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.885+0000 7f30f7a84640 1 -- 172.21.15.5:0/3482405105 >> 172.21.15.5:0/3482405105 conn(0x7f30e8005920 msgr2=0x7f30e8005d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:01.868 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.886+0000 7f30f7a84640 1 -- 172.21.15.5:0/3482405105 shutdown_connections 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.886+0000 7f30f7a84640 1 -- 172.21.15.5:0/3482405105 wait complete. 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.887+0000 7f30f7a84640 10 monclient: build_initial_monmap 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.887+0000 7f30f7a84640 10 monclient: monmap: 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:01.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.888+0000 7f30f7a84640 1 librados: starting msgr at 2024-03-29T10:55:01.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.888+0000 7f30f7a84640 1 librados: starting objecter 2024-03-29T10:55:01.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.888+0000 7f30f7a84640 1 Processor -- start 2024-03-29T10:55:01.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.888+0000 7f30f7a84640 1 -- start start 2024-03-29T10:55:01.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.888+0000 7f30f7a84640 1 librados: setting wanted keys 2024-03-29T10:55:01.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.888+0000 7f30f7a84640 1 librados: calling monclient init 2024-03-29T10:55:01.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.888+0000 7f30f7a84640 10 monclient: init 2024-03-29T10:55:01.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.889+0000 7f30f7a84640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:01.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.889+0000 7f30f7a84640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:01.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.889+0000 7f30f7a84640 10 monclient(hunting): picked mon.noname-a con 0x7f30e802cd20 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:01.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.889+0000 7f30f7a84640 10 monclient(hunting): picked mon.noname-b con 0x7f30e8035310 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:01.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.889+0000 7f30f7a84640 10 monclient(hunting): picked mon.noname-c con 0x7f30e8036ac0 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:01.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.889+0000 7f30f7a84640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e80c5540 con 0x7f30e802cd20 2024-03-29T10:55:01.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.889+0000 7f30f7a84640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e80bf1e0 con 0x7f30e8035310 2024-03-29T10:55:01.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.889+0000 7f30f7a84640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e80c7d60 con 0x7f30e8036ac0 2024-03-29T10:55:01.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.889+0000 7f30f7a84640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:01.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.889+0000 7f30f7a84640 10 monclient(hunting): authenticate will time out at 1315.130258s 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.891+0000 7f30f6021640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f30e8036ac0 0x7f30e8174120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:44922/0 (socket says 172.21.15.5:44922) 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.891+0000 7f30f6021640 1 -- 172.21.15.5:0/1594685655 learned_addr learned my addr 172.21.15.5:0/1594685655 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 10 client.?.objecter ms_handle_connect 0x7f30e8036ac0 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 10 client.?.objecter ms_handle_connect 0x7f30e8035310 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 10 client.?.objecter ms_handle_connect 0x7f30e802cd20 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752372826 0 0) 0x7f30e80bf1e0 con 0x7f30e8035310 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:01.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:01.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:55:01.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.892+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d4004eb0 con 0x7f30e8035310 2024-03-29T10:55:01.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577725390 0 0) 0x7f30e80c7d60 con 0x7f30e8036ac0 2024-03-29T10:55:01.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:01.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:01.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:01.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient(hunting): my global_id is 4541 2024-03-29T10:55:01.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e80bf1e0 con 0x7f30e8036ac0 2024-03-29T10:55:01.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 802439694 0 0) 0x7f30d4004eb0 con 0x7f30e8035310 2024-03-29T10:55:01.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e80c7d60 con 0x7f30e8035310 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f30e0001870 con 0x7f30e8035310 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient(hunting): dump: 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:01.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4135684677 0 0) 0x7f30e80c7d60 con 0x7f30e8035310 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 >> v1:172.21.15.5:6790/0 conn(0x7f30e8036ac0 legacy=0x7f30e8174120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 >> v1:172.21.15.5:6789/0 conn(0x7f30e802cd20 legacy=0x7f30e80c4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 monclient: found mon.b 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e80c8040 con 0x7f30e8035310 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30f7a84640 5 monclient: authenticate success, global_id 4426 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30f7a84640 10 monclient: _renew_subs 2024-03-29T10:55:01.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30f7a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30f7a84640 1 -- 172.21.15.5:0/1594685655 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e80be3b0 con 0x7f30e8035310 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30f7a84640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30f7a84640 10 monclient: _renew_subs 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30f7a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30f7a84640 1 -- 172.21.15.5:0/1594685655 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e80d3dc0 con 0x7f30e8035310 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30f7a84640 1 librados: init done 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30e00019d0 con 0x7f30e8035310 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f30e00037d0 con 0x7f30e8035310 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.893+0000 7f30e67fc640 10 monclient: dump: 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:01.880 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:01.881 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:01.881 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:01.881 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:01.881 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:01.881 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:01.881 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:01.881 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.894+0000 7f30e7fff640 4 set_mon_vals no callback set 2024-03-29T10:55:01.882 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.896+0000 7f30f7a84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.896+0000 7f30f7a84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.896+0000 7f30f7a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.896+0000 7f30f7a84640 1 -- 172.21.15.5:0/1594685655 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d8005080 con 0x7f30e8035310 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.896+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f30e00197a0 con 0x7f30e8035310 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.897+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f30e0083870 con 0x7f30e8035310 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.897+0000 7f30e67fc640 10 client.4426.objecter ms_dispatch 0x7f30f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.897+0000 7f30e67fc640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.897+0000 7f30e67fc640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.897+0000 7f30e67fc640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.898+0000 7f30f5820640 10 monclient: get_auth_request con 0x7f30d406dc40 auth_method 0 2024-03-29T10:55:01.883 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.898+0000 7f30e67fc640 10 client.4426.objecter ms_handle_connect 0x7f30d406dc40 2024-03-29T10:55:01.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.906+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30e004ef80 con 0x7f30e8035310 2024-03-29T10:55:01.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.906+0000 7f30e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:01.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:01.906+0000 7f30e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:02.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.234+0000 7f30f7a84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:02.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.234+0000 7f30f7a84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:02.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.234+0000 7f30f7a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:02.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.234+0000 7f30f7a84640 1 -- 172.21.15.5:0/1594685655 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30d8005330 con 0x7f30e8035310 2024-03-29T10:55:02.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.237+0000 7f30e67fc640 1 -- 172.21.15.5:0/1594685655 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f30e0051080 con 0x7f30e8035310 2024-03-29T10:55:02.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.237+0000 7f30e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:02.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.237+0000 7f30e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:02.218 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:55:02.218 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:02.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.240+0000 7f30f7a84640 10 librados: watch_flush enter 2024-03-29T10:55:02.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.240+0000 7f30f7a84640 10 librados: watch_flush exit 2024-03-29T10:55:02.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.240+0000 7f30f7a84640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T10:55:02.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.240+0000 7f30f7a84640 10 client.4426.objecter successfully canceled tick 2024-03-29T10:55:02.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.240+0000 7f30f7a84640 1 -- 172.21.15.5:0/1594685655 >> v1:172.21.15.5:6816/2847500151 conn(0x7f30d406dc40 legacy=0x7f30d4070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.240+0000 7f30f7a84640 10 monclient: shutdown 2024-03-29T10:55:02.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.240+0000 7f30f7a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:02.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.240+0000 7f30f7a84640 1 -- 172.21.15.5:0/1594685655 >> v1:172.21.15.177:6789/0 conn(0x7f30e8035310 legacy=0x7f30e80d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.241+0000 7f30f7a84640 1 -- 172.21.15.5:0/1594685655 shutdown_connections 2024-03-29T10:55:02.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.241+0000 7f30f7a84640 1 -- 172.21.15.5:0/1594685655 >> 172.21.15.5:0/1594685655 conn(0x7f30e8005920 msgr2=0x7f30e80c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:02.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.241+0000 7f30f7a84640 1 -- 172.21.15.5:0/1594685655 shutdown_connections 2024-03-29T10:55:02.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.241+0000 7f30f7a84640 1 -- 172.21.15.5:0/1594685655 wait complete. 2024-03-29T10:55:02.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:02.241+0000 7f30f7a84640 1 librados: shutdown 2024-03-29T10:55:02.238 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:09.044 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.212+0000 7f8013a63640 10 monclient: get_monmap_and_config 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.212+0000 7f8013a63640 10 monclient: build_initial_monmap 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.212+0000 7f8013a63640 10 monclient: monmap: 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:55:09.213436+0000 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:55:09.213436+0000 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:09.194 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:55:09.195 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:55:09.195 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:09.195 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.213+0000 7f8013a63640 1 Processor -- start 2024-03-29T10:55:09.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.214+0000 7f8013a63640 1 -- start start 2024-03-29T10:55:09.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.214+0000 7f8013a63640 10 monclient: init 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.214+0000 7f8013a63640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.214+0000 7f8013a63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.214+0000 7f8013a63640 10 monclient(hunting): picked mon.noname-b con 0x7f800c0b0770 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.214+0000 7f8013a63640 10 monclient(hunting): picked mon.noname-c con 0x7f800c14a700 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.215+0000 7f8013a63640 10 monclient(hunting): picked mon.noname-a con 0x7f800c0ac5a0 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.215+0000 7f8013a63640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c0aba70 con 0x7f800c0ac5a0 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.215+0000 7f8013a63640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c0aecb0 con 0x7f800c14a700 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.215+0000 7f8013a63640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c0af150 con 0x7f800c0b0770 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.215+0000 7f8013a63640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.215+0000 7f8013a63640 10 monclient(hunting): authenticate will time out at 1322.456136s 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.215+0000 7f80117ff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f800c0b0770 0x7f800c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:53286/0 (socket says 172.21.15.5:53286) 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.215+0000 7f80117ff640 1 -- 172.21.15.5:0/1387655442 learned_addr learned my addr 172.21.15.5:0/1387655442 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:09.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.216+0000 7f8003fff640 1 -- 172.21.15.5:0/1387655442 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612336739 0 0) 0x7f800c0af150 con 0x7f800c0b0770 2024-03-29T10:55:09.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.216+0000 7f8003fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.216+0000 7f8003fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.216+0000 7f8003fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.216+0000 7f8003fff640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:55:09.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.216+0000 7f8003fff640 1 -- 172.21.15.5:0/1387655442 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff8003520 con 0x7f800c0b0770 2024-03-29T10:55:09.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.217+0000 7f8003fff640 1 -- 172.21.15.5:0/1387655442 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1583742690 0 0) 0x7f7ff8003520 con 0x7f800c0b0770 2024-03-29T10:55:09.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.217+0000 7f8003fff640 1 -- 172.21.15.5:0/1387655442 >> v1:172.21.15.177:6789/0 conn(0x7f800c14a700 legacy=0x7f800c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:09.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.218+0000 7f8003fff640 1 -- 172.21.15.5:0/1387655442 >> v1:172.21.15.5:6789/0 conn(0x7f800c0ac5a0 legacy=0x7f800c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.220+0000 7f8003fff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.220+0000 7f8003fff640 1 monclient: found mon.noname-b 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.220+0000 7f8003fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.220+0000 7f8003fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 1 -- 172.21.15.5:0/1387655442 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c0ac9d0 con 0x7f800c0b0770 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 10 monclient: _finish_auth 0 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 1 -- 172.21.15.5:0/1387655442 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8008001690 con 0x7f800c0b0770 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 10 monclient: dump: 2024-03-29T10:55:09.202 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:09.203 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:09.203 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:09.203 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:09.203 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.203 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:09.203 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:09.203 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:09.203 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:09.203 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:09.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 1 -- 172.21.15.5:0/1387655442 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80080018f0 con 0x7f800c0b0770 2024-03-29T10:55:09.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:09.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 1 -- 172.21.15.5:0/1387655442 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8008001d40 con 0x7f800c0b0770 2024-03-29T10:55:09.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:09.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8003fff640 10 monclient: dump: 2024-03-29T10:55:09.204 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:09.204 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:09.204 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8013a63640 5 monclient: authenticate success, global_id 4544 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8013a63640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.221+0000 7f8013a63640 4 set_mon_vals no callback set 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.222+0000 7f8013a63640 10 monclient: shutdown 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.222+0000 7f8013a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:09.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.222+0000 7f8013a63640 1 -- 172.21.15.5:0/1387655442 >> v1:172.21.15.5:6790/0 conn(0x7f800c0b0770 legacy=0x7f800c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.223+0000 7f8013a63640 1 -- 172.21.15.5:0/1387655442 shutdown_connections 2024-03-29T10:55:09.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.223+0000 7f8013a63640 1 -- 172.21.15.5:0/1387655442 >> 172.21.15.5:0/1387655442 conn(0x7f800c06b660 msgr2=0x7f800c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:09.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.225+0000 7f8013a63640 1 -- 172.21.15.5:0/1387655442 shutdown_connections 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.225+0000 7f8013a63640 1 -- 172.21.15.5:0/1387655442 wait complete. 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.226+0000 7f8013a63640 10 monclient: build_initial_monmap 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.226+0000 7f8013a63640 10 monclient: monmap: 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.226+0000 7f8013a63640 1 librados: starting msgr at 2024-03-29T10:55:09.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.226+0000 7f8013a63640 1 librados: starting objecter 2024-03-29T10:55:09.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.226+0000 7f8013a63640 1 Processor -- start 2024-03-29T10:55:09.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.226+0000 7f8013a63640 1 -- start start 2024-03-29T10:55:09.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.226+0000 7f8013a63640 1 librados: setting wanted keys 2024-03-29T10:55:09.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.226+0000 7f8013a63640 1 librados: calling monclient init 2024-03-29T10:55:09.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.226+0000 7f8013a63640 10 monclient: init 2024-03-29T10:55:09.210 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.227+0000 7f8013a63640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:09.210 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.227+0000 7f8013a63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:09.210 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.227+0000 7f8013a63640 10 monclient(hunting): picked mon.noname-c con 0x7f800c0ac5a0 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:09.210 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.227+0000 7f8013a63640 10 monclient(hunting): picked mon.noname-b con 0x7f800c0b0770 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:09.210 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.227+0000 7f8013a63640 10 monclient(hunting): picked mon.noname-a con 0x7f800c14a700 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:09.210 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.227+0000 7f8013a63640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c06f460 con 0x7f800c14a700 2024-03-29T10:55:09.211 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.227+0000 7f8013a63640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c1db4c0 con 0x7f800c0b0770 2024-03-29T10:55:09.211 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.227+0000 7f8013a63640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c141ed0 con 0x7f800c0ac5a0 2024-03-29T10:55:09.211 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.227+0000 7f8013a63640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:09.211 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.227+0000 7f8013a63640 10 monclient(hunting): authenticate will time out at 1322.468697s 2024-03-29T10:55:09.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.228+0000 7f80117ff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f800c0ac5a0 0x7f800c06de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:57008/0 (socket says 172.21.15.5:57008) 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.228+0000 7f80117ff640 1 -- 172.21.15.5:0/252757094 learned_addr learned my addr 172.21.15.5:0/252757094 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.228+0000 7f8001ffb640 10 client.?.objecter ms_handle_connect 0x7f800c0ac5a0 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.228+0000 7f8001ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.228+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298493707 0 0) 0x7f800c141ed0 con 0x7f800c0ac5a0 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.228+0000 7f8001ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.228+0000 7f8001ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.228+0000 7f8001ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.228+0000 7f8001ffb640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.228+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff0004eb0 con 0x7f800c0ac5a0 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.231+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1875724216 0 0) 0x7f7ff0004eb0 con 0x7f800c0ac5a0 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f800c141ed0 con 0x7f800c0ac5a0 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8008001d40 con 0x7f800c0ac5a0 2024-03-29T10:55:09.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 client.?.objecter ms_handle_connect 0x7f800c14a700 2024-03-29T10:55:09.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099215315 0 0) 0x7f800c06f460 con 0x7f800c14a700 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 monclient(hunting): my global_id is 4464 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff0004eb0 con 0x7f800c14a700 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 client.?.objecter ms_handle_connect 0x7f800c0b0770 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.232+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1371607683 0 0) 0x7f800c141ed0 con 0x7f800c0ac5a0 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 >> v1:172.21.15.177:6789/0 conn(0x7f800c0b0770 legacy=0x7f800c1db9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 >> v1:172.21.15.5:6789/0 conn(0x7f800c14a700 legacy=0x7f800c13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8001ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8001ffb640 1 monclient: found mon.c 2024-03-29T10:55:09.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8001ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8001ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c142060 con 0x7f800c0ac5a0 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8001ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8001ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8001ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8013a63640 5 monclient: authenticate success, global_id 4547 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8013a63640 10 monclient: _renew_subs 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8013a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.233+0000 7f8013a63640 1 -- 172.21.15.5:0/252757094 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800c1dc0f0 con 0x7f800c0ac5a0 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.234+0000 7f8013a63640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.234+0000 7f8013a63640 10 monclient: _renew_subs 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.234+0000 7f8013a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.234+0000 7f8013a63640 1 -- 172.21.15.5:0/252757094 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c145210 con 0x7f800c0ac5a0 2024-03-29T10:55:09.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.234+0000 7f8013a63640 1 librados: init done 2024-03-29T10:55:09.221 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.235+0000 7f8013a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.235+0000 7f8013a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.235+0000 7f8013a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.235+0000 7f8013a63640 1 -- 172.21.15.5:0/252757094 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800c0ab560 con 0x7f800c0ac5a0 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.236+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8008004d20 con 0x7f800c0ac5a0 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.236+0000 7f8001ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.236+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8008005030 con 0x7f800c0ac5a0 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.236+0000 7f8001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.236+0000 7f8001ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.236+0000 7f8001ffb640 10 monclient: dump: 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:09.222 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.236+0000 7f8002ffd640 4 set_mon_vals no callback set 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.237+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f800801b000 con 0x7f800c0ac5a0 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.238+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f8008084e30 con 0x7f800c0ac5a0 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.238+0000 7f8001ffb640 10 client.4547.objecter ms_dispatch 0x7f800c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.238+0000 7f8010ffe640 10 monclient: get_auth_request con 0x7f7ff006ddc0 auth_method 0 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.238+0000 7f8001ffb640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.238+0000 7f8001ffb640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.238+0000 7f8001ffb640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T10:55:09.223 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.238+0000 7f8001ffb640 10 client.4547.objecter ms_handle_connect 0x7f7ff006ddc0 2024-03-29T10:55:09.227 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.246+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8008050710 con 0x7f800c0ac5a0 2024-03-29T10:55:09.227 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.246+0000 7f8001ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:09.227 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.246+0000 7f8001ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:09.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.577+0000 7f8013a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:09.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.577+0000 7f8013a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:09.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.577+0000 7f8013a63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:09.558 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.577+0000 7f8013a63640 1 -- 172.21.15.5:0/252757094 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f800c1db670 con 0x7f800c0ac5a0 2024-03-29T10:55:09.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.579+0000 7f8001ffb640 1 -- 172.21.15.5:0/252757094 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f800c1db670 con 0x7f800c0ac5a0 2024-03-29T10:55:09.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.579+0000 7f8001ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:09.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.579+0000 7f8001ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:09.561 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:55:09.561 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:09.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.583+0000 7f8013a63640 10 librados: watch_flush enter 2024-03-29T10:55:09.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.583+0000 7f8013a63640 10 librados: watch_flush exit 2024-03-29T10:55:09.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.583+0000 7f8013a63640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T10:55:09.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.583+0000 7f8013a63640 10 client.4547.objecter successfully canceled tick 2024-03-29T10:55:09.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.585+0000 7f8013a63640 1 -- 172.21.15.5:0/252757094 >> v1:172.21.15.5:6816/2847500151 conn(0x7f7ff006ddc0 legacy=0x7f7ff0070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.585+0000 7f8013a63640 10 monclient: shutdown 2024-03-29T10:55:09.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.585+0000 7f8013a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:09.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.585+0000 7f8013a63640 1 -- 172.21.15.5:0/252757094 >> v1:172.21.15.5:6790/0 conn(0x7f800c0ac5a0 legacy=0x7f800c06de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.585+0000 7f8013a63640 1 -- 172.21.15.5:0/252757094 shutdown_connections 2024-03-29T10:55:09.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.585+0000 7f8013a63640 1 -- 172.21.15.5:0/252757094 >> 172.21.15.5:0/252757094 conn(0x7f800c06b660 msgr2=0x7f800c149130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:09.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.585+0000 7f8013a63640 1 -- 172.21.15.5:0/252757094 shutdown_connections 2024-03-29T10:55:09.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.585+0000 7f8013a63640 1 -- 172.21.15.5:0/252757094 wait complete. 2024-03-29T10:55:09.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:09.585+0000 7f8013a63640 1 librados: shutdown 2024-03-29T10:55:09.582 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:16.388 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:16.546 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.565+0000 7f406c515640 10 monclient: get_monmap_and_config 2024-03-29T10:55:16.546 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.565+0000 7f406c515640 10 monclient: build_initial_monmap 2024-03-29T10:55:16.546 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.565+0000 7f406c515640 10 monclient: monmap: 2024-03-29T10:55:16.546 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:16.546 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:16.546 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:55:16.566270+0000 2024-03-29T10:55:16.546 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:55:16.566270+0000 2024-03-29T10:55:16.547 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:16.547 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:16.547 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:16.547 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:55:16.547 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:55:16.547 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:16.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.566+0000 7f406c515640 1 Processor -- start 2024-03-29T10:55:16.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.566+0000 7f406c515640 1 -- start start 2024-03-29T10:55:16.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.566+0000 7f406c515640 10 monclient: init 2024-03-29T10:55:16.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406c515640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:16.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406c515640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:16.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406c515640 10 monclient(hunting): picked mon.noname-b con 0x7f40640b4790 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:16.548 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406c515640 10 monclient(hunting): picked mon.noname-c con 0x7f40640aea90 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:16.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406c515640 10 monclient(hunting): picked mon.noname-a con 0x7f40640b39f0 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:16.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406c515640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40640b0c60 con 0x7f40640b39f0 2024-03-29T10:55:16.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406c515640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40640af020 con 0x7f40640aea90 2024-03-29T10:55:16.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406c515640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406413fbd0 con 0x7f40640b4790 2024-03-29T10:55:16.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406c515640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:16.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406c515640 10 monclient(hunting): authenticate will time out at 1329.808707s 2024-03-29T10:55:16.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406aab2640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f40640b39f0 0x7f406413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57622/0 (socket says 172.21.15.5:57622) 2024-03-29T10:55:16.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406aab2640 1 -- 172.21.15.5:0/807918320 learned_addr learned my addr 172.21.15.5:0/807918320 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:16.549 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.567+0000 7f406a2b1640 1 --1- 172.21.15.5:0/807918320 >> v1:172.21.15.5:6790/0 conn(0x7f40640b4790 0x7f40640b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:40932/0 (socket says 172.21.15.5:40932) 2024-03-29T10:55:16.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213029607 0 0) 0x7f406413fbd0 con 0x7f40640b4790 2024-03-29T10:55:16.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 10 monclient(hunting): _negotiate 2024-03-29T10:55:16.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:16.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:16.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:55:16.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4050003520 con 0x7f40640b4790 2024-03-29T10:55:16.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921460757 0 0) 0x7f40640b0c60 con 0x7f40640b39f0 2024-03-29T10:55:16.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 10 monclient(hunting): _negotiate 2024-03-29T10:55:16.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:16.551 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 10 monclient(hunting): my global_id is 4467 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406413fbd0 con 0x7f40640b39f0 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2190012309 0 0) 0x7f4050003520 con 0x7f40640b4790 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 >> v1:172.21.15.177:6789/0 conn(0x7f40640aea90 legacy=0x7f40640af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 >> v1:172.21.15.5:6789/0 conn(0x7f40640b39f0 legacy=0x7f406413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 10 monclient: _finish_hunting 0 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 1 monclient: found mon.noname-b 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.568+0000 7f40692af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406413fd60 con 0x7f40640b4790 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 10 monclient: _finish_auth 0 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 10 monclient: _check_auth_tickets 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:16.552 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4060001690 con 0x7f40640b4790 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 10 monclient: dump: 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:16.553 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f406c515640 5 monclient: authenticate success, global_id 4550 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f406c515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40600017f0 con 0x7f40640b4790 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 1 -- 172.21.15.5:0/807918320 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4060001c00 con 0x7f40640b4790 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f40692af640 10 monclient: dump: 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:16.554 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f406c515640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.569+0000 7f406c515640 4 set_mon_vals no callback set 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.570+0000 7f406c515640 10 monclient: shutdown 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.570+0000 7f406c515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.570+0000 7f406c515640 1 -- 172.21.15.5:0/807918320 >> v1:172.21.15.5:6790/0 conn(0x7f40640b4790 legacy=0x7f40640b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.570+0000 7f406c515640 1 -- 172.21.15.5:0/807918320 shutdown_connections 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.570+0000 7f406c515640 1 -- 172.21.15.5:0/807918320 >> 172.21.15.5:0/807918320 conn(0x7f4064066a00 msgr2=0x7f4064066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:16.555 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.570+0000 7f406c515640 1 -- 172.21.15.5:0/807918320 shutdown_connections 2024-03-29T10:55:16.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.570+0000 7f406c515640 1 -- 172.21.15.5:0/807918320 wait complete. 2024-03-29T10:55:16.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.571+0000 7f406c515640 10 monclient: build_initial_monmap 2024-03-29T10:55:16.559 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.571+0000 7f406c515640 10 monclient: monmap: 2024-03-29T10:55:16.559 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:16.559 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:16.559 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:55:16.559 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:55:16.559 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:16.559 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:16.559 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.571+0000 7f406c515640 1 librados: starting msgr at 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.571+0000 7f406c515640 1 librados: starting objecter 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.572+0000 7f406c515640 1 Processor -- start 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.572+0000 7f406c515640 1 -- start start 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.572+0000 7f406c515640 1 librados: setting wanted keys 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.572+0000 7f406c515640 1 librados: calling monclient init 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.572+0000 7f406c515640 10 monclient: init 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.572+0000 7f406c515640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406c515640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406c515640 10 monclient(hunting): picked mon.noname-b con 0x7f40640b39f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406c515640 10 monclient(hunting): picked mon.noname-c con 0x7f40640b4790 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406a2b1640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f40640b39f0 0x7f406406fa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:58810/0 (socket says 172.21.15.5:58810) 2024-03-29T10:55:16.560 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406aab2640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f406406f030 0x7f40641e62e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58012/0 (socket says 172.21.15.5:58012) 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f4069ab0640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f40640b4790 0x7f4064070170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:41580/0 (socket says 172.21.15.5:41580) 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406c515640 10 monclient(hunting): picked mon.noname-a con 0x7f406406f030 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406c515640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4064071e50 con 0x7f406406f030 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406c515640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4064071b20 con 0x7f40640b39f0 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406c515640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4064071140 con 0x7f40640b4790 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406c515640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406c515640 10 monclient(hunting): authenticate will time out at 1329.814622s 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f406a2b1640 1 -- 172.21.15.5:0/4251286008 learned_addr learned my addr 172.21.15.5:0/4251286008 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f405affd640 10 client.?.objecter ms_handle_connect 0x7f406406f030 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f405affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f405affd640 10 client.?.objecter ms_handle_connect 0x7f40640b39f0 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f405affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f405affd640 10 client.?.objecter ms_handle_connect 0x7f40640b4790 2024-03-29T10:55:16.561 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.573+0000 7f405affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352019009 0 0) 0x7f4064071140 con 0x7f40640b4790 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4048004eb0 con 0x7f40640b4790 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220585912 0 0) 0x7f4064071e50 con 0x7f406406f030 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): my global_id is 4470 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4064071140 con 0x7f406406f030 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2375341092 0 0) 0x7f4064071140 con 0x7f406406f030 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4064071e50 con 0x7f406406f030 2024-03-29T10:55:16.562 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f405c001cb0 con 0x7f406406f030 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): dump: 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:16.563 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1309676834 0 0) 0x7f4048004eb0 con 0x7f40640b4790 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4064071140 con 0x7f40640b4790 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4054001910 con 0x7f40640b4790 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.574+0000 7f405affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 10 monclient(hunting): dump: 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:16.564 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3629208485 0 0) 0x7f4064071e50 con 0x7f406406f030 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 >> v1:172.21.15.5:6790/0 conn(0x7f40640b4790 legacy=0x7f4064070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 >> v1:172.21.15.177:6789/0 conn(0x7f40640b39f0 legacy=0x7f406406fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 1 monclient: found mon.a 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064071420 con 0x7f406406f030 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 10 monclient: _finish_auth 0 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f406c515640 5 monclient: authenticate success, global_id 4470 2024-03-29T10:55:16.565 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f406c515640 10 monclient: _renew_subs 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f406c515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f406c515640 1 -- 172.21.15.5:0/4251286008 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4064071700 con 0x7f406406f030 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f406c515640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f406c515640 10 monclient: _renew_subs 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f406c515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f406c515640 1 -- 172.21.15.5:0/4251286008 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4064141ee0 con 0x7f406406f030 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f406c515640 1 librados: init done 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f405c004b60 con 0x7f406406f030 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f405affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.575+0000 7f4068aae640 4 set_mon_vals no callback set 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.576+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f405c004f30 con 0x7f406406f030 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.576+0000 7f405affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:16.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.576+0000 7f405affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.576+0000 7f405affd640 10 monclient: dump: 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:16.567 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:16.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.576+0000 7f406c515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:16.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.576+0000 7f406c515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:16.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.576+0000 7f406c515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:55:16.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.576+0000 7f406c515640 1 -- 172.21.15.5:0/4251286008 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404c005080 con 0x7f406406f030 2024-03-29T10:55:16.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.578+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f405c005090 con 0x7f406406f030 2024-03-29T10:55:16.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.579+0000 7f406a2b1640 10 monclient: get_auth_request con 0x7f404806df10 auth_method 0 2024-03-29T10:55:16.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.579+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f405c084ca0 con 0x7f406406f030 2024-03-29T10:55:16.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.579+0000 7f405affd640 10 client.4470.objecter ms_dispatch 0x7f4064000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:16.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.579+0000 7f405affd640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:16.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.579+0000 7f405affd640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:16.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.579+0000 7f405affd640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-29T10:55:16.569 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.579+0000 7f405affd640 10 client.4470.objecter ms_handle_connect 0x7f404806df10 2024-03-29T10:55:16.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.594+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f405c0505e0 con 0x7f406406f030 2024-03-29T10:55:16.574 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.594+0000 7f405affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:16.575 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.594+0000 7f405affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:16.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.931+0000 7f406c515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:16.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.931+0000 7f406c515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:16.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.931+0000 7f406c515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:55:16.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.931+0000 7f406c515640 1 -- 172.21.15.5:0/4251286008 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f404c005330 con 0x7f406406f030 2024-03-29T10:55:16.913 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:55:16.914 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:16.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.932+0000 7f405affd640 1 -- 172.21.15.5:0/4251286008 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f405c084350 con 0x7f406406f030 2024-03-29T10:55:16.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.932+0000 7f405affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:16.914 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.932+0000 7f405affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:16.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 10 librados: watch_flush enter 2024-03-29T10:55:16.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 10 librados: watch_flush exit 2024-03-29T10:55:16.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-29T10:55:16.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 10 client.4470.objecter successfully canceled tick 2024-03-29T10:55:16.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 1 -- 172.21.15.5:0/4251286008 >> v1:172.21.15.5:6816/2847500151 conn(0x7f404806df10 legacy=0x7f40480703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 10 monclient: shutdown 2024-03-29T10:55:16.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:16.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 1 -- 172.21.15.5:0/4251286008 >> v1:172.21.15.5:6789/0 conn(0x7f406406f030 legacy=0x7f40641e62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:16.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 1 -- 172.21.15.5:0/4251286008 shutdown_connections 2024-03-29T10:55:16.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 1 -- 172.21.15.5:0/4251286008 >> 172.21.15.5:0/4251286008 conn(0x7f4064066a00 msgr2=0x7f406413e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:16.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 1 -- 172.21.15.5:0/4251286008 shutdown_connections 2024-03-29T10:55:16.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 1 -- 172.21.15.5:0/4251286008 wait complete. 2024-03-29T10:55:16.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:16.936+0000 7f406c515640 1 librados: shutdown 2024-03-29T10:55:16.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:23.741 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:23.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.922+0000 7fdf41965640 10 monclient: get_monmap_and_config 2024-03-29T10:55:23.903 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.922+0000 7fdf41965640 10 monclient: build_initial_monmap 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.922+0000 7fdf41965640 10 monclient: monmap: 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:55:23.923412+0000 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:55:23.923412+0000 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:55:23.904 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:23.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.925+0000 7fdf41965640 1 Processor -- start 2024-03-29T10:55:23.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.925+0000 7fdf41965640 1 -- start start 2024-03-29T10:55:23.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.925+0000 7fdf41965640 10 monclient: init 2024-03-29T10:55:23.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.926+0000 7fdf41965640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:23.907 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.926+0000 7fdf41965640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:23.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.926+0000 7fdf41965640 10 monclient(hunting): picked mon.noname-c con 0x7fdf3c0aac30 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:23.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.926+0000 7fdf41965640 10 monclient(hunting): picked mon.noname-b con 0x7fdf3c072f20 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:23.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.926+0000 7fdf41965640 10 monclient(hunting): picked mon.noname-a con 0x7fdf3c0a9e90 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:23.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.926+0000 7fdf41965640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3c0ad8e0 con 0x7fdf3c0a9e90 2024-03-29T10:55:23.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.926+0000 7fdf41965640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3c0ac6a0 con 0x7fdf3c0aac30 2024-03-29T10:55:23.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.926+0000 7fdf41965640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3c0ab230 con 0x7fdf3c072f20 2024-03-29T10:55:23.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.926+0000 7fdf41965640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:23.908 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.926+0000 7fdf41965640 10 monclient(hunting): authenticate will time out at 1337.167534s 2024-03-29T10:55:23.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.931+0000 7fdf41164640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fdf3c0a9e90 0x7fdf3c0ad1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33051/0 (socket says 172.21.15.5:33051) 2024-03-29T10:55:23.912 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.931+0000 7fdf41164640 1 -- 172.21.15.5:0/3598947569 learned_addr learned my addr 172.21.15.5:0/3598947569 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:23.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.932+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461578279 0 0) 0x7fdf3c0ad8e0 con 0x7fdf3c0a9e90 2024-03-29T10:55:23.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.932+0000 7fdf3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.932+0000 7fdf3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.932+0000 7fdf3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.932+0000 7fdf3b7fe640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:55:23.913 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.932+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf2c003520 con 0x7fdf3c0a9e90 2024-03-29T10:55:23.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667602571 0 0) 0x7fdf3c0ac6a0 con 0x7fdf3c0aac30 2024-03-29T10:55:23.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:55:23.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf3c0ad8e0 con 0x7fdf3c0aac30 2024-03-29T10:55:23.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1825027505 0 0) 0x7fdf2c003520 con 0x7fdf3c0a9e90 2024-03-29T10:55:23.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 >> v1:172.21.15.5:6790/0 conn(0x7fdf3c072f20 legacy=0x7fdf3c0acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.916 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 >> v1:172.21.15.177:6789/0 conn(0x7fdf3c0aac30 legacy=0x7fdf3c13af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 1 monclient: found mon.noname-a 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c13b660 con 0x7fdf3c0a9e90 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf41965640 5 monclient: authenticate success, global_id 4473 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.933+0000 7fdf41965640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.934+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fdf34005010 con 0x7fdf3c0a9e90 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.934+0000 7fdf3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.934+0000 7fdf3b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.934+0000 7fdf3b7fe640 10 monclient: dump: 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:23.917 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:23.918 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:23.918 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:23.918 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.918 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:23.918 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:23.918 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:23.918 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:23.918 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:23.918 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.934+0000 7fdf41965640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:23.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.936+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf34005230 con 0x7fdf3c0a9e90 2024-03-29T10:55:23.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.936+0000 7fdf3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:23.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.936+0000 7fdf3b7fe640 1 -- 172.21.15.5:0/3598947569 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fdf34001790 con 0x7fdf3c0a9e90 2024-03-29T10:55:23.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.936+0000 7fdf3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.936+0000 7fdf3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:23.919 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.936+0000 7fdf3b7fe640 10 monclient: dump: 2024-03-29T10:55:23.919 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:23.919 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:23.919 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:23.919 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.936+0000 7fdf41965640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.936+0000 7fdf41965640 4 set_mon_vals no callback set 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.937+0000 7fdf41965640 10 monclient: shutdown 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.937+0000 7fdf41965640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.937+0000 7fdf41965640 1 -- 172.21.15.5:0/3598947569 >> v1:172.21.15.5:6789/0 conn(0x7fdf3c0a9e90 legacy=0x7fdf3c0ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.938+0000 7fdf41965640 1 -- 172.21.15.5:0/3598947569 shutdown_connections 2024-03-29T10:55:23.920 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.938+0000 7fdf41965640 1 -- 172.21.15.5:0/3598947569 >> 172.21.15.5:0/3598947569 conn(0x7fdf3c0b3a30 msgr2=0x7fdf3c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:23.921 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.939+0000 7fdf41965640 1 -- 172.21.15.5:0/3598947569 shutdown_connections 2024-03-29T10:55:23.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.940+0000 7fdf41965640 1 -- 172.21.15.5:0/3598947569 wait complete. 2024-03-29T10:55:23.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.941+0000 7fdf41965640 10 monclient: build_initial_monmap 2024-03-29T10:55:23.923 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.941+0000 7fdf41965640 10 monclient: monmap: 2024-03-29T10:55:23.923 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:23.923 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:23.923 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:55:23.923 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.942+0000 7fdf41965640 1 librados: starting msgr at 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.942+0000 7fdf41965640 1 librados: starting objecter 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.942+0000 7fdf41965640 1 Processor -- start 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.942+0000 7fdf41965640 1 -- start start 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.942+0000 7fdf41965640 1 librados: setting wanted keys 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.942+0000 7fdf41965640 1 librados: calling monclient init 2024-03-29T10:55:23.924 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.942+0000 7fdf41965640 10 monclient: init 2024-03-29T10:55:23.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.943+0000 7fdf41965640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:23.925 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.943+0000 7fdf41965640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:23.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.946+0000 7fdf41965640 10 monclient(hunting): picked mon.noname-b con 0x7fdf3c072f20 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:23.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.946+0000 7fdf41965640 10 monclient(hunting): picked mon.noname-a con 0x7fdf3c0a9e90 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:23.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.946+0000 7fdf40963640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fdf3c072f20 0x7fdf3c12d990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:59783/0 (socket says 172.21.15.5:59783) 2024-03-29T10:55:23.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.946+0000 7fdf40963640 1 -- 172.21.15.5:0/2369096981 learned_addr learned my addr 172.21.15.5:0/2369096981 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:23.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.947+0000 7fdf41965640 10 monclient(hunting): picked mon.noname-c con 0x7fdf3c0aac30 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:23.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.947+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3c12b310 con 0x7fdf3c0a9e90 2024-03-29T10:55:23.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.947+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3c149f70 con 0x7fdf3c072f20 2024-03-29T10:55:23.928 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.947+0000 7fdf397fa640 10 client.?.objecter ms_handle_connect 0x7fdf3c072f20 2024-03-29T10:55:23.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.947+0000 7fdf397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:23.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.947+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3c12ef80 con 0x7fdf3c0aac30 2024-03-29T10:55:23.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.947+0000 7fdf41965640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:23.929 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.947+0000 7fdf41965640 10 monclient(hunting): authenticate will time out at 1337.188238s 2024-03-29T10:55:23.937 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375067963 0 0) 0x7fdf3c149f70 con 0x7fdf3c072f20 2024-03-29T10:55:23.937 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.937 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.937 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.937 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:55:23.937 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf20004eb0 con 0x7fdf3c072f20 2024-03-29T10:55:23.937 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 10 client.?.objecter ms_handle_connect 0x7fdf3c0aac30 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065417035 0 0) 0x7fdf3c12ef80 con 0x7fdf3c0aac30 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.951+0000 7fdf397fa640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf3c149f70 con 0x7fdf3c0aac30 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1589521141 0 0) 0x7fdf20004eb0 con 0x7fdf3c072f20 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf3c12ef80 con 0x7fdf3c072f20 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fdf28001970 con 0x7fdf3c072f20 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient(hunting): dump: 2024-03-29T10:55:23.938 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1946782623 0 0) 0x7fdf3c149f70 con 0x7fdf3c0aac30 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf20004eb0 con 0x7fdf3c0aac30 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fdf34001600 con 0x7fdf3c0aac30 2024-03-29T10:55:23.939 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient(hunting): dump: 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 client.?.objecter ms_handle_connect 0x7fdf3c0a9e90 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3600189489 0 0) 0x7fdf3c12ef80 con 0x7fdf3c072f20 2024-03-29T10:55:23.940 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 >> v1:172.21.15.5:6790/0 conn(0x7fdf3c0aac30 legacy=0x7fdf3c12ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 >> v1:172.21.15.5:6789/0 conn(0x7fdf3c0a9e90 legacy=0x7fdf3c12a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient: _finish_hunting 0 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 monclient: found mon.b 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c12f260 con 0x7fdf3c072f20 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient: _finish_auth 0 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient: _check_auth_tickets 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf41965640 5 monclient: authenticate success, global_id 4438 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf41965640 10 monclient: _renew_subs 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf41965640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3c12e0a0 con 0x7fdf3c072f20 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf41965640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:23.941 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf41965640 10 monclient: _renew_subs 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf41965640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3c12f860 con 0x7fdf3c072f20 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf41965640 1 librados: init done 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf28003610 con 0x7fdf3c072f20 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fdf28003960 con 0x7fdf3c072f20 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf397fa640 10 monclient: dump: 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:23.942 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.952+0000 7fdf3affd640 4 set_mon_vals no callback set 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.955+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fdf28003ac0 con 0x7fdf3c072f20 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.955+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fdf280849f0 con 0x7fdf3c072f20 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.955+0000 7fdf397fa640 10 client.4438.objecter ms_dispatch 0x7fdf3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.955+0000 7fdf397fa640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.955+0000 7fdf397fa640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.955+0000 7fdf397fa640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.956+0000 7fdf3bfff640 10 monclient: get_auth_request con 0x7fdf2006e130 auth_method 0 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.956+0000 7fdf41965640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:23.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.956+0000 7fdf41965640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:23.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.956+0000 7fdf41965640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:23.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.956+0000 7fdf397fa640 10 client.4438.objecter ms_handle_connect 0x7fdf2006e130 2024-03-29T10:55:23.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.956+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c005080 con 0x7fdf3c072f20 2024-03-29T10:55:23.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.965+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf280502d0 con 0x7fdf3c072f20 2024-03-29T10:55:23.947 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.965+0000 7fdf397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:23.947 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:23.965+0000 7fdf397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:24.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.311+0000 7fdf41965640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:24.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.311+0000 7fdf41965640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:24.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.311+0000 7fdf41965640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:24.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.311+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf1c005330 con 0x7fdf3c072f20 2024-03-29T10:55:24.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.312+0000 7fdf397fa640 1 -- 172.21.15.5:0/2369096981 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fdf28052200 con 0x7fdf3c072f20 2024-03-29T10:55:24.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.312+0000 7fdf397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:24.294 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:55:24.294 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:24.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.312+0000 7fdf397fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:24.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.316+0000 7fdf41965640 10 librados: watch_flush enter 2024-03-29T10:55:24.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.316+0000 7fdf41965640 10 librados: watch_flush exit 2024-03-29T10:55:24.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.316+0000 7fdf41965640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-29T10:55:24.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.316+0000 7fdf41965640 10 client.4438.objecter successfully canceled tick 2024-03-29T10:55:24.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.316+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 >> v1:172.21.15.5:6816/2847500151 conn(0x7fdf2006e130 legacy=0x7fdf200705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.316+0000 7fdf41965640 10 monclient: shutdown 2024-03-29T10:55:24.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.316+0000 7fdf41965640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:24.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.316+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 >> v1:172.21.15.177:6789/0 conn(0x7fdf3c072f20 legacy=0x7fdf3c12d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.316+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 shutdown_connections 2024-03-29T10:55:24.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.316+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 >> 172.21.15.5:0/2369096981 conn(0x7fdf3c0b3a30 msgr2=0x7fdf3c0aefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:24.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.317+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 shutdown_connections 2024-03-29T10:55:24.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.317+0000 7fdf41965640 1 -- 172.21.15.5:0/2369096981 wait complete. 2024-03-29T10:55:24.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:24.317+0000 7fdf41965640 1 librados: shutdown 2024-03-29T10:55:24.313 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:31.120 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:31.276 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.295+0000 7f20757bc640 10 monclient: get_monmap_and_config 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.295+0000 7f20757bc640 10 monclient: build_initial_monmap 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.295+0000 7f20757bc640 10 monclient: monmap: 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:55:31.296518+0000 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:55:31.296518+0000 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:55:31.277 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:55:31.278 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:31.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.296+0000 7f20757bc640 1 Processor -- start 2024-03-29T10:55:31.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.296+0000 7f20757bc640 1 -- start start 2024-03-29T10:55:31.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.296+0000 7f20757bc640 10 monclient: init 2024-03-29T10:55:31.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.297+0000 7f20757bc640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:31.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.297+0000 7f20757bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:31.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.297+0000 7f20757bc640 10 monclient(hunting): picked mon.noname-a con 0x7f20700a9750 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:31.279 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.297+0000 7f20757bc640 10 monclient(hunting): picked mon.noname-b con 0x7f20700add40 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:31.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.297+0000 7f20757bc640 10 monclient(hunting): picked mon.noname-c con 0x7f20700aad20 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:31.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.297+0000 7f20757bc640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2070149870 con 0x7f20700a9750 2024-03-29T10:55:31.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.297+0000 7f20757bc640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20700abf80 con 0x7f20700aad20 2024-03-29T10:55:31.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.297+0000 7f20757bc640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2070149ac0 con 0x7f20700add40 2024-03-29T10:55:31.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.297+0000 7f20757bc640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:31.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.297+0000 7f20757bc640 10 monclient(hunting): authenticate will time out at 1344.539040s 2024-03-29T10:55:31.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.301+0000 7f206f7fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f20700add40 0x7f2070143610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:60796/0 (socket says 172.21.15.5:60796) 2024-03-29T10:55:31.282 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.302+0000 7f206f7fe640 1 -- 172.21.15.5:0/869504398 learned_addr learned my addr 172.21.15.5:0/869504398 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:31.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700725037 0 0) 0x7f2070149ac0 con 0x7f20700add40 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205c003520 con 0x7f20700add40 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754443607 0 0) 0x7f20700abf80 con 0x7f20700aad20 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2070149ac0 con 0x7f20700aad20 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342162288 0 0) 0x7f2070149870 con 0x7f20700a9750 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:55:31.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.305+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20700abf80 con 0x7f20700a9750 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1876460467 0 0) 0x7f205c003520 con 0x7f20700add40 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 >> v1:172.21.15.177:6789/0 conn(0x7f20700aad20 legacy=0x7f2070151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 >> v1:172.21.15.5:6789/0 conn(0x7f20700a9750 legacy=0x7f2070149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 1 monclient: found mon.noname-b 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207014a6f0 con 0x7f20700add40 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 10 monclient: _finish_auth 0 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:31.291 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2060001a10 con 0x7f20700add40 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f206effd640 10 monclient: dump: 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f20757bc640 5 monclient: authenticate success, global_id 4562 2024-03-29T10:55:31.292 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.306+0000 7f20757bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:31.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.310+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2060001b70 con 0x7f20700add40 2024-03-29T10:55:31.293 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.310+0000 7f206effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:31.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.310+0000 7f20757bc640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:31.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.310+0000 7f20757bc640 4 set_mon_vals no callback set 2024-03-29T10:55:31.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.310+0000 7f206effd640 1 -- 172.21.15.5:0/869504398 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2060001fc0 con 0x7f20700add40 2024-03-29T10:55:31.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.310+0000 7f206effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.310+0000 7f206effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:31.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.310+0000 7f206effd640 10 monclient: dump: 2024-03-29T10:55:31.294 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:31.294 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.311+0000 7f20757bc640 10 monclient: shutdown 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.311+0000 7f20757bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.311+0000 7f20757bc640 1 -- 172.21.15.5:0/869504398 >> v1:172.21.15.5:6790/0 conn(0x7f20700add40 legacy=0x7f2070143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.311+0000 7f20757bc640 1 -- 172.21.15.5:0/869504398 shutdown_connections 2024-03-29T10:55:31.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.311+0000 7f20757bc640 1 -- 172.21.15.5:0/869504398 >> 172.21.15.5:0/869504398 conn(0x7f20700673e0 msgr2=0x7f20700b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:31.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.314+0000 7f20757bc640 1 -- 172.21.15.5:0/869504398 shutdown_connections 2024-03-29T10:55:31.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.315+0000 7f20757bc640 1 -- 172.21.15.5:0/869504398 wait complete. 2024-03-29T10:55:31.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.316+0000 7f20757bc640 10 monclient: build_initial_monmap 2024-03-29T10:55:31.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.316+0000 7f20757bc640 10 monclient: monmap: 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.317+0000 7f20757bc640 1 librados: starting msgr at 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.317+0000 7f20757bc640 1 librados: starting objecter 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.317+0000 7f20757bc640 1 Processor -- start 2024-03-29T10:55:31.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.317+0000 7f20757bc640 1 -- start start 2024-03-29T10:55:31.300 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.317+0000 7f20757bc640 1 librados: setting wanted keys 2024-03-29T10:55:31.300 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.317+0000 7f20757bc640 1 librados: calling monclient init 2024-03-29T10:55:31.300 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.317+0000 7f20757bc640 10 monclient: init 2024-03-29T10:55:31.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.318+0000 7f20757bc640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:31.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.318+0000 7f20757bc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:31.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.318+0000 7f20757bc640 10 monclient(hunting): picked mon.noname-a con 0x7f20700a9750 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:31.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.318+0000 7f20757bc640 10 monclient(hunting): picked mon.noname-b con 0x7f20700aad20 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:31.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.319+0000 7f20757bc640 10 monclient(hunting): picked mon.noname-c con 0x7f20700add40 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:31.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.319+0000 7f20757bc640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207006e420 con 0x7f20700a9750 2024-03-29T10:55:31.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.319+0000 7f20757bc640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2070147150 con 0x7f20700aad20 2024-03-29T10:55:31.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.319+0000 7f20757bc640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207006e690 con 0x7f20700add40 2024-03-29T10:55:31.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.319+0000 7f20757bc640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:31.302 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.319+0000 7f20757bc640 10 monclient(hunting): authenticate will time out at 1344.560466s 2024-03-29T10:55:31.302 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f2074fbb640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f20700add40 0x7f207013c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:33328/0 (socket says 172.21.15.5:33328) 2024-03-29T10:55:31.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f2074fbb640 1 -- 172.21.15.5:0/2673048465 learned_addr learned my addr 172.21.15.5:0/2673048465 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:31.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f206cff9640 10 client.?.objecter ms_handle_connect 0x7f20700add40 2024-03-29T10:55:31.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f206cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:31.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f206cff9640 10 client.?.objecter ms_handle_connect 0x7f20700a9750 2024-03-29T10:55:31.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f206cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:31.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173206019 0 0) 0x7f207006e420 con 0x7f20700a9750 2024-03-29T10:55:31.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f206cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f206cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f206cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f206cff9640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:55:31.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.321+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2050004eb0 con 0x7f20700a9750 2024-03-29T10:55:31.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811408185 0 0) 0x7f207006e690 con 0x7f20700add40 2024-03-29T10:55:31.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:55:31.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f207006e420 con 0x7f20700add40 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1925426843 0 0) 0x7f2050004eb0 con 0x7f20700a9750 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f207006e690 con 0x7f20700a9750 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2058001870 con 0x7f20700a9750 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 10 monclient(hunting): dump: 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:31.308 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.322+0000 7f206cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 174919299 0 0) 0x7f207006e690 con 0x7f20700a9750 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 >> v1:172.21.15.5:6790/0 conn(0x7f20700add40 legacy=0x7f207013c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 >> v1:172.21.15.177:6789/0 conn(0x7f20700aad20 legacy=0x7f2070146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 1 monclient: found mon.a 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20700b0550 con 0x7f20700a9750 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:31.309 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f20757bc640 5 monclient: authenticate success, global_id 4479 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f20757bc640 10 monclient: _renew_subs 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f20757bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2058003610 con 0x7f20700a9750 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.323+0000 7f20757bc640 1 -- 172.21.15.5:0/2673048465 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20700b07a0 con 0x7f20700a9750 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.324+0000 7f206cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.324+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2058001d80 con 0x7f20700a9750 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.324+0000 7f206cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.324+0000 7f206cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.324+0000 7f206cff9640 10 monclient: dump: 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:31.310 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.325+0000 7f20757bc640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.325+0000 7f20757bc640 10 monclient: _renew_subs 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.325+0000 7f20757bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.325+0000 7f20757bc640 1 -- 172.21.15.5:0/2673048465 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2070144350 con 0x7f20700a9750 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.325+0000 7f20757bc640 1 librados: init done 2024-03-29T10:55:31.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.327+0000 7f206e7fc640 4 set_mon_vals no callback set 2024-03-29T10:55:31.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.327+0000 7f20757bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:31.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.328+0000 7f20757bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:31.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.328+0000 7f20757bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:55:31.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.328+0000 7f20757bc640 1 -- 172.21.15.5:0/2673048465 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2054005080 con 0x7f20700a9750 2024-03-29T10:55:31.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.330+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f20580039e0 con 0x7f20700a9750 2024-03-29T10:55:31.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.331+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f2058083790 con 0x7f20700a9750 2024-03-29T10:55:31.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.331+0000 7f206cff9640 10 client.4479.objecter ms_dispatch 0x7f2070000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:31.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.331+0000 7f206cff9640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:31.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.331+0000 7f206cff9640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:31.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.331+0000 7f206cff9640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-29T10:55:31.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.331+0000 7f206f7fe640 10 monclient: get_auth_request con 0x7f205006dd30 auth_method 0 2024-03-29T10:55:31.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.332+0000 7f206cff9640 10 client.4479.objecter ms_handle_connect 0x7f205006dd30 2024-03-29T10:55:31.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.340+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f205804eea0 con 0x7f20700a9750 2024-03-29T10:55:31.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.340+0000 7f206cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:31.322 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.340+0000 7f206cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:31.691 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.709+0000 7f20757bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:31.691 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.709+0000 7f20757bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:31.691 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.709+0000 7f20757bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:55:31.691 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.710+0000 7f20757bc640 1 -- 172.21.15.5:0/2673048465 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2054005330 con 0x7f20700a9750 2024-03-29T10:55:31.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.714+0000 7f206cff9640 1 -- 172.21.15.5:0/2673048465 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f2058050fa0 con 0x7f20700a9750 2024-03-29T10:55:31.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.714+0000 7f206cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:31.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.714+0000 7f206cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:31.696 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:55:31.696 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:31.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.718+0000 7f20757bc640 10 librados: watch_flush enter 2024-03-29T10:55:31.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.718+0000 7f20757bc640 10 librados: watch_flush exit 2024-03-29T10:55:31.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.718+0000 7f20757bc640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-29T10:55:31.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.718+0000 7f20757bc640 10 client.4479.objecter successfully canceled tick 2024-03-29T10:55:31.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.719+0000 7f20757bc640 1 -- 172.21.15.5:0/2673048465 >> v1:172.21.15.5:6816/2847500151 conn(0x7f205006dd30 legacy=0x7f20500701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.719+0000 7f20757bc640 10 monclient: shutdown 2024-03-29T10:55:31.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.719+0000 7f20757bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:31.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.719+0000 7f20757bc640 1 -- 172.21.15.5:0/2673048465 >> v1:172.21.15.5:6789/0 conn(0x7f20700a9750 legacy=0x7f2070145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.719+0000 7f20757bc640 1 -- 172.21.15.5:0/2673048465 shutdown_connections 2024-03-29T10:55:31.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.719+0000 7f20757bc640 1 -- 172.21.15.5:0/2673048465 >> 172.21.15.5:0/2673048465 conn(0x7f20700673e0 msgr2=0x7f2070140650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:31.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.719+0000 7f20757bc640 1 -- 172.21.15.5:0/2673048465 shutdown_connections 2024-03-29T10:55:31.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.719+0000 7f20757bc640 1 -- 172.21.15.5:0/2673048465 wait complete. 2024-03-29T10:55:31.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:31.723+0000 7f20757bc640 1 librados: shutdown 2024-03-29T10:55:31.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:38.527 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:38.679 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.698+0000 7f9df76d6640 10 monclient: get_monmap_and_config 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.698+0000 7f9df76d6640 10 monclient: build_initial_monmap 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.698+0000 7f9df76d6640 10 monclient: monmap: 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:55:38.699806+0000 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:55:38.699806+0000 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:55:38.680 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:38.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.699+0000 7f9df76d6640 1 Processor -- start 2024-03-29T10:55:38.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.700+0000 7f9df76d6640 1 -- start start 2024-03-29T10:55:38.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.700+0000 7f9df76d6640 10 monclient: init 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df76d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df76d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df76d6640 10 monclient(hunting): picked mon.noname-c con 0x7f9df01457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df76d6640 10 monclient(hunting): picked mon.noname-b con 0x7f9df013f650 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df76d6640 10 monclient(hunting): picked mon.noname-a con 0x7f9df00a9d70 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df76d6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df00aaef0 con 0x7f9df00a9d70 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df76d6640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df0145bd0 con 0x7f9df01457f0 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df76d6640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df0145eb0 con 0x7f9df013f650 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df76d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df76d6640 10 monclient(hunting): authenticate will time out at 1351.942240s 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df5472640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f9df01457f0 0x7f9df013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:38914/0 (socket says 172.21.15.5:38914) 2024-03-29T10:55:38.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9df5472640 1 -- 172.21.15.5:0/2713041659 learned_addr learned my addr 172.21.15.5:0/2713041659 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099194227 0 0) 0x7f9df0145bd0 con 0x7f9df01457f0 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9de7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9de7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9de7fff640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.701+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ddc003520 con 0x7f9df01457f0 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515954932 0 0) 0x7f9df0145eb0 con 0x7f9df013f650 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df0145bd0 con 0x7f9df013f650 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289833708 0 0) 0x7f9df00aaef0 con 0x7f9df00a9d70 2024-03-29T10:55:38.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient(hunting): my global_id is 4482 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df0145eb0 con 0x7f9df00a9d70 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3863899832 0 0) 0x7f9df0145eb0 con 0x7f9df00a9d70 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 >> v1:172.21.15.5:6790/0 conn(0x7f9df013f650 legacy=0x7f9df00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 >> v1:172.21.15.177:6789/0 conn(0x7f9df01457f0 legacy=0x7f9df013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 1 monclient: found mon.noname-a 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df0146340 con 0x7f9df00a9d70 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient: _finish_auth 0 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:38.685 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9de8001930 con 0x7f9df00a9d70 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9de7fff640 10 monclient: dump: 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:38.686 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9df76d6640 5 monclient: authenticate success, global_id 4482 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.702+0000 7f9df76d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.703+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9de8001a90 con 0x7f9df00a9d70 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.703+0000 7f9de7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.703+0000 7f9de7fff640 1 -- 172.21.15.5:0/2713041659 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9de8001ee0 con 0x7f9df00a9d70 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.703+0000 7f9de7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.703+0000 7f9de7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.703+0000 7f9de7fff640 10 monclient: dump: 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:38.687 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:38.688 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:38.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.703+0000 7f9df76d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:38.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.703+0000 7f9df76d6640 4 set_mon_vals no callback set 2024-03-29T10:55:38.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.704+0000 7f9df76d6640 10 monclient: shutdown 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.704+0000 7f9df76d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.704+0000 7f9df76d6640 1 -- 172.21.15.5:0/2713041659 >> v1:172.21.15.5:6789/0 conn(0x7f9df00a9d70 legacy=0x7f9df014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.704+0000 7f9df76d6640 1 -- 172.21.15.5:0/2713041659 shutdown_connections 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.704+0000 7f9df76d6640 1 -- 172.21.15.5:0/2713041659 >> 172.21.15.5:0/2713041659 conn(0x7f9df0066a00 msgr2=0x7f9df0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.704+0000 7f9df76d6640 1 -- 172.21.15.5:0/2713041659 shutdown_connections 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.705+0000 7f9df76d6640 1 -- 172.21.15.5:0/2713041659 wait complete. 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.705+0000 7f9df76d6640 10 monclient: build_initial_monmap 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.705+0000 7f9df76d6640 10 monclient: monmap: 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:55:38.694 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.706+0000 7f9df76d6640 1 librados: starting msgr at 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.706+0000 7f9df76d6640 1 librados: starting objecter 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.706+0000 7f9df76d6640 1 Processor -- start 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.706+0000 7f9df76d6640 1 -- start start 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.706+0000 7f9df76d6640 1 librados: setting wanted keys 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.706+0000 7f9df76d6640 1 librados: calling monclient init 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.706+0000 7f9df76d6640 10 monclient: init 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df76d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df76d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df76d6640 10 monclient(hunting): picked mon.noname-b con 0x7f9df00a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:38.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df76d6640 10 monclient(hunting): picked mon.noname-a con 0x7f9df013f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df76d6640 10 monclient(hunting): picked mon.noname-c con 0x7f9df01457f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df76d6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df0140280 con 0x7f9df013f650 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df76d6640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df0141a10 con 0x7f9df00a9d70 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df76d6640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df0141c80 con 0x7f9df01457f0 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df76d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df76d6640 10 monclient(hunting): authenticate will time out at 1351.948486s 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df5c73640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f9df01457f0 0x7f9df01e1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:58912/0 (socket says 172.21.15.5:58912) 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df4c71640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9df013f650 0x7f9df01426f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53444/0 (socket says 172.21.15.5:53444) 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9df5c73640 1 -- 172.21.15.5:0/1166432878 learned_addr learned my addr 172.21.15.5:0/1166432878 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9de5ffb640 10 client.?.objecter ms_handle_connect 0x7f9df013f650 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.707+0000 7f9de5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 10 client.?.objecter ms_handle_connect 0x7f9df01457f0 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219107696 0 0) 0x7f9df0141c80 con 0x7f9df01457f0 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd4004eb0 con 0x7f9df01457f0 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190113654 0 0) 0x7f9df0140280 con 0x7f9df013f650 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.708+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df0141c80 con 0x7f9df013f650 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 client.?.objecter ms_handle_connect 0x7f9df00a9d70 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1269949941 0 0) 0x7f9dd4004eb0 con 0x7f9df01457f0 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dd4005490 con 0x7f9df01457f0 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9de8001e60 con 0x7f9df01457f0 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043424421 0 0) 0x7f9df0141a10 con 0x7f9df00a9d70 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:38.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd4004eb0 con 0x7f9df00a9d70 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3205451485 0 0) 0x7f9df0141c80 con 0x7f9df013f650 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9df0141a10 con 0x7f9df013f650 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9de0001930 con 0x7f9df013f650 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.699 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1401620813 0 0) 0x7f9dd4004eb0 con 0x7f9df00a9d70 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dd4006150 con 0x7f9df00a9d70 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9dec001710 con 0x7f9df00a9d70 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:38.700 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3914559336 0 0) 0x7f9dd4005490 con 0x7f9df01457f0 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.709+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 >> v1:172.21.15.177:6789/0 conn(0x7f9df00a9d70 legacy=0x7f9df0140710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 >> v1:172.21.15.5:6789/0 conn(0x7f9df013f650 legacy=0x7f9df01426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 1 monclient: found mon.c 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:38.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df0142e00 con 0x7f9df01457f0 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9df76d6640 5 monclient: authenticate success, global_id 4571 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9df76d6640 10 monclient: _renew_subs 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9df76d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9df76d6640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df013fda0 con 0x7f9df01457f0 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9de8003c20 con 0x7f9df01457f0 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f9de8004030 con 0x7f9df01457f0 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de5ffb640 10 monclient: dump: 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:38.702 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.710+0000 7f9de77fe640 4 set_mon_vals no callback set 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.711+0000 7f9df76d6640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.711+0000 7f9df76d6640 10 monclient: _renew_subs 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.711+0000 7f9df76d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.711+0000 7f9df76d6640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df0138f10 con 0x7f9df01457f0 2024-03-29T10:55:38.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.711+0000 7f9df76d6640 1 librados: init done 2024-03-29T10:55:38.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.712+0000 7f9df76d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:38.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.713+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f9de8004190 con 0x7f9df01457f0 2024-03-29T10:55:38.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.713+0000 7f9df76d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:38.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.714+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f9de8054be0 con 0x7f9df01457f0 2024-03-29T10:55:38.705 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.714+0000 7f9de5ffb640 10 client.4571.objecter ms_dispatch 0x7f9df0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:38.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.714+0000 7f9de5ffb640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:38.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.714+0000 7f9de5ffb640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:38.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.714+0000 7f9de5ffb640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T10:55:38.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.713+0000 7f9df76d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:38.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.714+0000 7f9df76d6640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df0140430 con 0x7f9df01457f0 2024-03-29T10:55:38.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.714+0000 7f9df5472640 10 monclient: get_auth_request con 0x7f9dd406e650 auth_method 0 2024-03-29T10:55:38.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.714+0000 7f9de5ffb640 10 client.4571.objecter ms_handle_connect 0x7f9dd406e650 2024-03-29T10:55:38.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.725+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9df0140430 con 0x7f9df01457f0 2024-03-29T10:55:38.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.725+0000 7f9de5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:38.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:38.725+0000 7f9de5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:39.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.132+0000 7f9df76d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:39.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.132+0000 7f9df76d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:39.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.132+0000 7f9df76d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:39.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.132+0000 7f9df76d6640 1 -- 172.21.15.5:0/1166432878 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9df0140430 con 0x7f9df01457f0 2024-03-29T10:55:39.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.133+0000 7f9de5ffb640 1 -- 172.21.15.5:0/1166432878 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f9df0140430 con 0x7f9df01457f0 2024-03-29T10:55:39.114 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.133+0000 7f9de5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:39.114 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.133+0000 7f9de5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:39.114 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:55:39.114 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:39.117 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.136+0000 7f9df76d6640 10 librados: watch_flush enter 2024-03-29T10:55:39.117 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.136+0000 7f9df76d6640 10 librados: watch_flush exit 2024-03-29T10:55:39.117 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.136+0000 7f9df76d6640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T10:55:39.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.136+0000 7f9df76d6640 10 client.4571.objecter successfully canceled tick 2024-03-29T10:55:39.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.136+0000 7f9df76d6640 1 -- 172.21.15.5:0/1166432878 >> v1:172.21.15.5:6816/2847500151 conn(0x7f9dd406e650 legacy=0x7f9dd4070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:39.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.136+0000 7f9df76d6640 10 monclient: shutdown 2024-03-29T10:55:39.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.136+0000 7f9df76d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:39.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.136+0000 7f9df76d6640 1 -- 172.21.15.5:0/1166432878 >> v1:172.21.15.5:6790/0 conn(0x7f9df01457f0 legacy=0x7f9df01e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:39.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.137+0000 7f9df76d6640 1 -- 172.21.15.5:0/1166432878 shutdown_connections 2024-03-29T10:55:39.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.137+0000 7f9df76d6640 1 -- 172.21.15.5:0/1166432878 >> 172.21.15.5:0/1166432878 conn(0x7f9df0066a00 msgr2=0x7f9df00b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:39.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.137+0000 7f9df76d6640 1 -- 172.21.15.5:0/1166432878 shutdown_connections 2024-03-29T10:55:39.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.137+0000 7f9df76d6640 1 -- 172.21.15.5:0/1166432878 wait complete. 2024-03-29T10:55:39.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:39.137+0000 7f9df76d6640 1 librados: shutdown 2024-03-29T10:55:39.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:45.940 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:46.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.103+0000 7f16d72ff640 10 monclient: get_monmap_and_config 2024-03-29T10:55:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.103+0000 7f16d72ff640 10 monclient: build_initial_monmap 2024-03-29T10:55:46.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.103+0000 7f16d72ff640 10 monclient: monmap: 2024-03-29T10:55:46.085 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:46.085 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:46.085 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:55:46.104317+0000 2024-03-29T10:55:46.085 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:55:46.104317+0000 2024-03-29T10:55:46.086 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:46.086 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:46.086 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:46.086 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:55:46.086 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:55:46.086 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:46.086 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.105+0000 7f16d72ff640 1 Processor -- start 2024-03-29T10:55:46.086 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.105+0000 7f16d72ff640 1 -- start start 2024-03-29T10:55:46.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.105+0000 7f16d72ff640 10 monclient: init 2024-03-29T10:55:46.088 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.106+0000 7f16d72ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:46.088 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.106+0000 7f16d72ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:46.088 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.106+0000 7f16d72ff640 10 monclient(hunting): picked mon.noname-b con 0x7f16d00b0500 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:46.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.106+0000 7f16d72ff640 10 monclient(hunting): picked mon.noname-c con 0x7f16d00b08e0 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:46.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.106+0000 7f16d72ff640 10 monclient(hunting): picked mon.noname-a con 0x7f16d00aeca0 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:46.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.106+0000 7f16d72ff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d0140620 con 0x7f16d00aeca0 2024-03-29T10:55:46.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.106+0000 7f16d72ff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d013daf0 con 0x7f16d00b08e0 2024-03-29T10:55:46.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.106+0000 7f16d72ff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d00ae180 con 0x7f16d00b0500 2024-03-29T10:55:46.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.106+0000 7f16d72ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:46.089 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.106+0000 7f16d72ff640 10 monclient(hunting): authenticate will time out at 1359.347500s 2024-03-29T10:55:46.090 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.109+0000 7f16d489a640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f16d00b08e0 0x7f16d013d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:41308/0 (socket says 172.21.15.5:41308) 2024-03-29T10:55:46.090 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.109+0000 7f16d489a640 1 -- 172.21.15.5:0/3895832165 learned_addr learned my addr 172.21.15.5:0/3895832165 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:46.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 1 -- 172.21.15.5:0/3895832165 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667282194 0 0) 0x7f16d00ae180 con 0x7f16d00b0500 2024-03-29T10:55:46.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:46.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:46.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:46.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:55:46.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 1 -- 172.21.15.5:0/3895832165 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16bc003520 con 0x7f16d00b0500 2024-03-29T10:55:46.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 1 -- 172.21.15.5:0/3895832165 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3222490915 0 0) 0x7f16bc003520 con 0x7f16d00b0500 2024-03-29T10:55:46.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 1 -- 172.21.15.5:0/3895832165 >> v1:172.21.15.177:6789/0 conn(0x7f16d00b08e0 legacy=0x7f16d013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:46.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 1 -- 172.21.15.5:0/3895832165 >> v1:172.21.15.5:6789/0 conn(0x7f16d00aeca0 legacy=0x7f16d014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:46.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 1 monclient: found mon.noname-b 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 1 -- 172.21.15.5:0/3895832165 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d0072510 con 0x7f16d00b0500 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient: _finish_auth 0 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 1 -- 172.21.15.5:0/3895832165 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f16c0001670 con 0x7f16d00b0500 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient: dump: 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:46.095 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 1 -- 172.21.15.5:0/3895832165 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16c0001850 con 0x7f16d00b0500 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.110+0000 7f16c7fff640 1 -- 172.21.15.5:0/3895832165 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f16c0001670 con 0x7f16d00b0500 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.111+0000 7f16c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.111+0000 7f16c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.111+0000 7f16c7fff640 10 monclient: dump: 2024-03-29T10:55:46.096 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.111+0000 7f16d72ff640 5 monclient: authenticate success, global_id 4574 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.111+0000 7f16d72ff640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.111+0000 7f16d72ff640 4 set_mon_vals no callback set 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.112+0000 7f16d72ff640 10 monclient: shutdown 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.112+0000 7f16d72ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:46.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.112+0000 7f16d72ff640 1 -- 172.21.15.5:0/3895832165 >> v1:172.21.15.5:6790/0 conn(0x7f16d00b0500 legacy=0x7f16d013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:46.101 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.114+0000 7f16d72ff640 1 -- 172.21.15.5:0/3895832165 shutdown_connections 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.114+0000 7f16d72ff640 1 -- 172.21.15.5:0/3895832165 >> 172.21.15.5:0/3895832165 conn(0x7f16d006b660 msgr2=0x7f16d006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.114+0000 7f16d72ff640 1 -- 172.21.15.5:0/3895832165 shutdown_connections 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.115+0000 7f16d72ff640 1 -- 172.21.15.5:0/3895832165 wait complete. 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.115+0000 7f16d72ff640 10 monclient: build_initial_monmap 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.115+0000 7f16d72ff640 10 monclient: monmap: 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:46.102 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.116+0000 7f16d72ff640 1 librados: starting msgr at 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.116+0000 7f16d72ff640 1 librados: starting objecter 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.116+0000 7f16d72ff640 1 Processor -- start 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.116+0000 7f16d72ff640 1 -- start start 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.116+0000 7f16d72ff640 1 librados: setting wanted keys 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.116+0000 7f16d72ff640 1 librados: calling monclient init 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.116+0000 7f16d72ff640 10 monclient: init 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.117+0000 7f16d72ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.117+0000 7f16d72ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.117+0000 7f16d72ff640 10 monclient(hunting): picked mon.noname-a con 0x7f16d00aeca0 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.117+0000 7f16d72ff640 10 monclient(hunting): picked mon.noname-c con 0x7f16d00b0500 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:46.103 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.117+0000 7f16d72ff640 10 monclient(hunting): picked mon.noname-b con 0x7f16d014b2f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.117+0000 7f16d72ff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d014ba10 con 0x7f16d00aeca0 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.117+0000 7f16d72ff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d006e460 con 0x7f16d014b2f0 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.117+0000 7f16d72ff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d006e6d0 con 0x7f16d00b0500 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.117+0000 7f16d72ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.117+0000 7f16d72ff640 10 monclient(hunting): authenticate will time out at 1359.358396s 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16d589c640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f16d014b2f0 0x7f16d01e62f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:41476/0 (socket says 172.21.15.5:41476) 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16d589c640 1 -- 172.21.15.5:0/1972504570 learned_addr learned my addr 172.21.15.5:0/1972504570 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 10 client.?.objecter ms_handle_connect 0x7f16d014b2f0 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977417773 0 0) 0x7f16d006e460 con 0x7f16d014b2f0 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:55:46.104 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b4004eb0 con 0x7f16d014b2f0 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3177602541 0 0) 0x7f16b4004eb0 con 0x7f16d014b2f0 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16d006e460 con 0x7f16d014b2f0 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f16c8001d00 con 0x7f16d014b2f0 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:46.105 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:46.106 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:46.106 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.118+0000 7f16c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:46.106 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.119+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2893757115 0 0) 0x7f16d006e460 con 0x7f16d014b2f0 2024-03-29T10:55:46.106 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.119+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 >> v1:172.21.15.5:6790/0 conn(0x7f16d00b0500 legacy=0x7f16d0133d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:46.106 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.120+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 >> v1:172.21.15.5:6789/0 conn(0x7f16d00aeca0 legacy=0x7f16d006eaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:46.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:46.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16c5ffb640 1 monclient: found mon.b 2024-03-29T10:55:46.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:46.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16c5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:46.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d014ae30 con 0x7f16d014b2f0 2024-03-29T10:55:46.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16c5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:46.108 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16d72ff640 5 monclient: authenticate success, global_id 4453 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16d72ff640 10 monclient: _renew_subs 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16d72ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.123+0000 7f16d72ff640 1 -- 172.21.15.5:0/1972504570 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d014a630 con 0x7f16d014b2f0 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.124+0000 7f16d72ff640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.124+0000 7f16d72ff640 10 monclient: _renew_subs 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.124+0000 7f16d72ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.124+0000 7f16d72ff640 1 -- 172.21.15.5:0/1972504570 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d014c250 con 0x7f16d014b2f0 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.124+0000 7f16d72ff640 1 librados: init done 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.125+0000 7f16d72ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.125+0000 7f16d72ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.125+0000 7f16d72ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:46.109 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.125+0000 7f16d72ff640 1 -- 172.21.15.5:0/1972504570 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d00ad0c0 con 0x7f16d014b2f0 2024-03-29T10:55:46.110 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.128+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16c8002000 con 0x7f16d014b2f0 2024-03-29T10:55:46.110 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.128+0000 7f16c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.128+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f16c8003de0 con 0x7f16d014b2f0 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.128+0000 7f16c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.128+0000 7f16c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.128+0000 7f16c5ffb640 10 monclient: dump: 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:46.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.128+0000 7f16c77fe640 4 set_mon_vals no callback set 2024-03-29T10:55:46.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.130+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f16c8004000 con 0x7f16d014b2f0 2024-03-29T10:55:46.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.131+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f16c8084df0 con 0x7f16d014b2f0 2024-03-29T10:55:46.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.131+0000 7f16c5ffb640 10 client.4453.objecter ms_dispatch 0x7f16d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:46.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.131+0000 7f16c5ffb640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:46.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.131+0000 7f16c5ffb640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:46.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.131+0000 7f16c5ffb640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T10:55:46.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.131+0000 7f16d509b640 10 monclient: get_auth_request con 0x7f16b406dbc0 auth_method 0 2024-03-29T10:55:46.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.132+0000 7f16c5ffb640 10 client.4453.objecter ms_handle_connect 0x7f16b406dbc0 2024-03-29T10:55:46.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.134+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16c80506d0 con 0x7f16d014b2f0 2024-03-29T10:55:46.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.134+0000 7f16c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:46.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.134+0000 7f16c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:46.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.469+0000 7f16d72ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:46.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.469+0000 7f16d72ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:46.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.469+0000 7f16d72ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:55:46.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.469+0000 7f16d72ff640 1 -- 172.21.15.5:0/1972504570 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16d0075f10 con 0x7f16d014b2f0 2024-03-29T10:55:46.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.470+0000 7f16c5ffb640 1 -- 172.21.15.5:0/1972504570 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f16c8052600 con 0x7f16d014b2f0 2024-03-29T10:55:46.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.470+0000 7f16c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:46.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.470+0000 7f16c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:46.453 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:55:46.453 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:46.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.474+0000 7f16d72ff640 10 librados: watch_flush enter 2024-03-29T10:55:46.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.474+0000 7f16d72ff640 10 librados: watch_flush exit 2024-03-29T10:55:46.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.474+0000 7f16d72ff640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T10:55:46.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.474+0000 7f16d72ff640 10 client.4453.objecter successfully canceled tick 2024-03-29T10:55:46.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.474+0000 7f16d72ff640 1 -- 172.21.15.5:0/1972504570 >> v1:172.21.15.5:6816/2847500151 conn(0x7f16b406dbc0 legacy=0x7f16b4070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:46.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.474+0000 7f16d72ff640 10 monclient: shutdown 2024-03-29T10:55:46.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.474+0000 7f16d72ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:46.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.474+0000 7f16d72ff640 1 -- 172.21.15.5:0/1972504570 >> v1:172.21.15.177:6789/0 conn(0x7f16d014b2f0 legacy=0x7f16d01e62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:46.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.475+0000 7f16d72ff640 1 -- 172.21.15.5:0/1972504570 shutdown_connections 2024-03-29T10:55:46.457 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.475+0000 7f16d72ff640 1 -- 172.21.15.5:0/1972504570 >> 172.21.15.5:0/1972504570 conn(0x7f16d006b660 msgr2=0x7f16d00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:46.457 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.475+0000 7f16d72ff640 1 -- 172.21.15.5:0/1972504570 shutdown_connections 2024-03-29T10:55:46.457 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.476+0000 7f16d72ff640 1 -- 172.21.15.5:0/1972504570 wait complete. 2024-03-29T10:55:46.457 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:46.476+0000 7f16d72ff640 1 librados: shutdown 2024-03-29T10:55:46.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:53.280 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:53.412 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.431+0000 7f0587d3e640 10 monclient: get_monmap_and_config 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.431+0000 7f0587d3e640 10 monclient: build_initial_monmap 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.431+0000 7f0587d3e640 10 monclient: monmap: 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:55:53.432933+0000 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:55:53.432933+0000 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:55:53.413 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:53.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.432+0000 7f0587d3e640 1 Processor -- start 2024-03-29T10:55:53.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.433+0000 7f0587d3e640 1 -- start start 2024-03-29T10:55:53.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.433+0000 7f0587d3e640 10 monclient: init 2024-03-29T10:55:53.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.433+0000 7f0587d3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:53.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.433+0000 7f0587d3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:53.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.433+0000 7f0587d3e640 10 monclient(hunting): picked mon.noname-b con 0x7f0580071de0 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:53.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.433+0000 7f0587d3e640 10 monclient(hunting): picked mon.noname-a con 0x7f05800721c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:53.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.433+0000 7f0587d3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0580145ae0 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:53.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.433+0000 7f0585ada640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f0580071de0 0x7f05800aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:53410/0 (socket says 172.21.15.5:53410) 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0585ada640 1 -- 172.21.15.5:0/3979757685 learned_addr learned my addr 172.21.15.5:0/3979757685 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.433+0000 7f0587d3e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0580145650 con 0x7f05800721c0 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0587d3e640 1 -- 172.21.15.5:0/3979757685 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05800ae260 con 0x7f0580145ae0 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0587d3e640 1 -- 172.21.15.5:0/3979757685 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05801420d0 con 0x7f0580071de0 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0587d3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0587d3e640 10 monclient(hunting): authenticate will time out at 1366.675332s 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0584ad8640 1 -- 172.21.15.5:0/3979757685 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170465414 0 0) 0x7f05801420d0 con 0x7f0580071de0 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0584ad8640 10 monclient(hunting): _negotiate 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0584ad8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0584ad8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0584ad8640 10 monclient(hunting): my global_id is 4577 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0584ad8640 1 -- 172.21.15.5:0/3979757685 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056c003520 con 0x7f0580071de0 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0584ad8640 1 -- 172.21.15.5:0/3979757685 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1011645975 0 0) 0x7f056c003520 con 0x7f0580071de0 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0584ad8640 1 -- 172.21.15.5:0/3979757685 >> v1:172.21.15.177:6789/0 conn(0x7f0580145ae0 legacy=0x7f0580145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:53.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.434+0000 7f0584ad8640 1 -- 172.21.15.5:0/3979757685 >> v1:172.21.15.5:6789/0 conn(0x7f05800721c0 legacy=0x7f058013f400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.436+0000 7f0584ad8640 10 monclient: _finish_hunting 0 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.436+0000 7f0584ad8640 1 monclient: found mon.noname-b 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.436+0000 7f0584ad8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 1 -- 172.21.15.5:0/3979757685 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05800b0590 con 0x7f0580071de0 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 10 monclient: _finish_auth 0 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 10 monclient: _check_auth_tickets 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 1 -- 172.21.15.5:0/3979757685 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f057c001690 con 0x7f0580071de0 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:53.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 10 monclient: dump: 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 1 -- 172.21.15.5:0/3979757685 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f057c0018b0 con 0x7f0580071de0 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0587d3e640 5 monclient: authenticate success, global_id 4577 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0587d3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 1 -- 172.21.15.5:0/3979757685 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f057c001c40 con 0x7f0580071de0 2024-03-29T10:55:53.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0584ad8640 10 monclient: dump: 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0587d3e640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:53.420 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.437+0000 7f0587d3e640 4 set_mon_vals no callback set 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.438+0000 7f0587d3e640 10 monclient: shutdown 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.438+0000 7f0587d3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.438+0000 7f0587d3e640 1 -- 172.21.15.5:0/3979757685 >> v1:172.21.15.5:6790/0 conn(0x7f0580071de0 legacy=0x7f05800aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.438+0000 7f0587d3e640 1 -- 172.21.15.5:0/3979757685 shutdown_connections 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.438+0000 7f0587d3e640 1 -- 172.21.15.5:0/3979757685 >> 172.21.15.5:0/3979757685 conn(0x7f058006b660 msgr2=0x7f058006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.438+0000 7f0587d3e640 1 -- 172.21.15.5:0/3979757685 shutdown_connections 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.438+0000 7f0587d3e640 1 -- 172.21.15.5:0/3979757685 wait complete. 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.439+0000 7f0587d3e640 10 monclient: build_initial_monmap 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.439+0000 7f0587d3e640 10 monclient: monmap: 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:53.428 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.440+0000 7f0587d3e640 1 librados: starting msgr at 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.440+0000 7f0587d3e640 1 librados: starting objecter 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.440+0000 7f0587d3e640 1 Processor -- start 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.440+0000 7f0587d3e640 1 -- start start 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.440+0000 7f0587d3e640 1 librados: setting wanted keys 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.440+0000 7f0587d3e640 1 librados: calling monclient init 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.440+0000 7f0587d3e640 10 monclient: init 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.440+0000 7f0587d3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.440+0000 7f0587d3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.441+0000 7f0587d3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0580071de0 addr v1:172.21.15.5:6790/0 2024-03-29T10:55:53.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.441+0000 7f0587d3e640 10 monclient(hunting): picked mon.noname-a con 0x7f0580145ae0 addr v1:172.21.15.5:6789/0 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.441+0000 7f0587d3e640 10 monclient(hunting): picked mon.noname-b con 0x7f05801d4b70 addr v1:172.21.15.177:6789/0 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.441+0000 7f0587d3e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05801d71b0 con 0x7f0580145ae0 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.441+0000 7f0587d3e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0580134970 con 0x7f05801d4b70 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.441+0000 7f0587d3e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05801d8030 con 0x7f0580071de0 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.441+0000 7f0587d3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.441+0000 7f0587d3e640 10 monclient(hunting): authenticate will time out at 1366.682225s 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.441+0000 7f0585ada640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f0580071de0 0x7f05801d7920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:54854/0 (socket says 172.21.15.5:54854) 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.441+0000 7f0585ada640 1 -- 172.21.15.5:0/311733182 learned_addr learned my addr 172.21.15.5:0/311733182 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 client.?.objecter ms_handle_connect 0x7f0580071de0 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 client.?.objecter ms_handle_connect 0x7f05801d4b70 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273701813 0 0) 0x7f05801d8030 con 0x7f0580071de0 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:53.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0564004eb0 con 0x7f0580071de0 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736926867 0 0) 0x7f0580134970 con 0x7f05801d4b70 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05801d8030 con 0x7f05801d4b70 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1941315992 0 0) 0x7f0564004eb0 con 0x7f0580071de0 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0580134970 con 0x7f0580071de0 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f057c001610 con 0x7f0580071de0 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): dump: 2024-03-29T10:55:53.431 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2094642744 0 0) 0x7f0580134970 con 0x7f0580071de0 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 >> v1:172.21.15.177:6789/0 conn(0x7f05801d4b70 legacy=0x7f0580132bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 >> v1:172.21.15.5:6789/0 conn(0x7f0580145ae0 legacy=0x7f05801d4460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 1 monclient: found mon.c 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.442+0000 7f05767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:53.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f05767fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05801d7400 con 0x7f0580071de0 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f05767fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f05767fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f05767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f0587d3e640 5 monclient: authenticate success, global_id 4580 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f0587d3e640 10 monclient: _renew_subs 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f0587d3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f0587d3e640 1 -- 172.21.15.5:0/311733182 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05801d3cb0 con 0x7f0580071de0 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f0587d3e640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f0587d3e640 10 monclient: _renew_subs 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f0587d3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f0587d3e640 1 -- 172.21.15.5:0/311733182 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0580133300 con 0x7f0580071de0 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f0587d3e640 1 librados: init done 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.443+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f057c004a90 con 0x7f0580071de0 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.444+0000 7f05767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.444+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f057c004da0 con 0x7f0580071de0 2024-03-29T10:55:53.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.444+0000 7f0577fff640 4 set_mon_vals no callback set 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.444+0000 7f05767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.444+0000 7f05767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.444+0000 7f05767fc640 10 monclient: dump: 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.444+0000 7f0587d3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:53.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.445+0000 7f0587d3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:53.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.445+0000 7f0587d3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:53.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.445+0000 7f0587d3e640 1 -- 172.21.15.5:0/311733182 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0568005080 con 0x7f0580071de0 2024-03-29T10:55:53.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.445+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f057c01ad70 con 0x7f0580071de0 2024-03-29T10:55:53.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.446+0000 7f05852d9640 10 monclient: get_auth_request con 0x7f056406dd90 auth_method 0 2024-03-29T10:55:53.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.446+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f057c084bf0 con 0x7f0580071de0 2024-03-29T10:55:53.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.446+0000 7f05767fc640 10 client.4580.objecter ms_dispatch 0x7f0580000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:53.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.446+0000 7f05767fc640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:53.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.446+0000 7f05767fc640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:53.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.446+0000 7f05767fc640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T10:55:53.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.446+0000 7f05767fc640 10 client.4580.objecter ms_handle_connect 0x7f056406dd90 2024-03-29T10:55:53.436 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.453+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f057c0504d0 con 0x7f0580071de0 2024-03-29T10:55:53.436 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.454+0000 7f05767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:53.436 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.454+0000 7f05767fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:53.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.769+0000 7f0587d3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:53.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.769+0000 7f0587d3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:53.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.769+0000 7f0587d3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:55:53.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.769+0000 7f0587d3e640 1 -- 172.21.15.5:0/311733182 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0568005330 con 0x7f0580071de0 2024-03-29T10:55:53.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.769+0000 7f05767fc640 1 -- 172.21.15.5:0/311733182 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f057c052400 con 0x7f0580071de0 2024-03-29T10:55:53.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.770+0000 7f05767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:53.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.770+0000 7f05767fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:53.751 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:55:53.751 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:53.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 10 librados: watch_flush enter 2024-03-29T10:55:53.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 10 librados: watch_flush exit 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 10 client.4580.objecter successfully canceled tick 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 1 -- 172.21.15.5:0/311733182 >> v1:172.21.15.5:6816/2847500151 conn(0x7f056406dd90 legacy=0x7f0564070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 10 monclient: shutdown 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 1 -- 172.21.15.5:0/311733182 >> v1:172.21.15.5:6790/0 conn(0x7f0580071de0 legacy=0x7f05801d7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 1 -- 172.21.15.5:0/311733182 shutdown_connections 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 1 -- 172.21.15.5:0/311733182 >> 172.21.15.5:0/311733182 conn(0x7f058006b660 msgr2=0x7f0580145f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 1 -- 172.21.15.5:0/311733182 shutdown_connections 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 1 -- 172.21.15.5:0/311733182 wait complete. 2024-03-29T10:55:53.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:55:53.773+0000 7f0587d3e640 1 librados: shutdown 2024-03-29T10:55:53.768 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:00.575 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:00.729 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.746+0000 7f77a30d9640 10 monclient: get_monmap_and_config 2024-03-29T10:56:00.729 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.746+0000 7f77a30d9640 10 monclient: build_initial_monmap 2024-03-29T10:56:00.729 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.746+0000 7f77a30d9640 10 monclient: monmap: 2024-03-29T10:56:00.729 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:00.729 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:00.729 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:56:00.747058+0000 2024-03-29T10:56:00.729 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:56:00.747058+0000 2024-03-29T10:56:00.729 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.747+0000 7f77a30d9640 1 Processor -- start 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.747+0000 7f77a30d9640 1 -- start start 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.747+0000 7f77a30d9640 10 monclient: init 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.747+0000 7f77a30d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.747+0000 7f77a30d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.747+0000 7f77a30d9640 10 monclient(hunting): picked mon.noname-a con 0x7f779c0ad6c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.748+0000 7f77a30d9640 10 monclient(hunting): picked mon.noname-b con 0x7f779c0af6c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:00.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.748+0000 7f77a30d9640 10 monclient(hunting): picked mon.noname-c con 0x7f779c14fdc0 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:00.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.748+0000 7f77a30d9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c0a94e0 con 0x7f779c0ad6c0 2024-03-29T10:56:00.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.748+0000 7f77a30d9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c0af3a0 con 0x7f779c14fdc0 2024-03-29T10:56:00.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.748+0000 7f77a30d9640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c0afb60 con 0x7f779c0af6c0 2024-03-29T10:56:00.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.748+0000 7f77a30d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:00.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.748+0000 7f77a30d9640 10 monclient(hunting): authenticate will time out at 1373.989188s 2024-03-29T10:56:00.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.748+0000 7f779bfff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f779c0af6c0 0x7f779c152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:41506/0 (socket says 172.21.15.5:41506) 2024-03-29T10:56:00.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.748+0000 7f779bfff640 1 -- 172.21.15.5:0/1522070182 learned_addr learned my addr 172.21.15.5:0/1522070182 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:00.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.749+0000 7f779b7fe640 1 -- 172.21.15.5:0/1522070182 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688266467 0 0) 0x7f779c0afb60 con 0x7f779c0af6c0 2024-03-29T10:56:00.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.749+0000 7f779b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:00.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.749+0000 7f779b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:00.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.749+0000 7f779b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:00.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.749+0000 7f779b7fe640 10 monclient(hunting): my global_id is 4583 2024-03-29T10:56:00.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.749+0000 7f779b7fe640 1 -- 172.21.15.5:0/1522070182 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7784003520 con 0x7f779c0af6c0 2024-03-29T10:56:00.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.749+0000 7f779b7fe640 1 -- 172.21.15.5:0/1522070182 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 906026602 0 0) 0x7f7784003520 con 0x7f779c0af6c0 2024-03-29T10:56:00.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.749+0000 7f779b7fe640 1 -- 172.21.15.5:0/1522070182 >> v1:172.21.15.177:6789/0 conn(0x7f779c14fdc0 legacy=0x7f779c15a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:00.732 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 1 -- 172.21.15.5:0/1522070182 >> v1:172.21.15.5:6789/0 conn(0x7f779c0ad6c0 legacy=0x7f779c14f6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:00.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:00.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 1 monclient: found mon.noname-b 2024-03-29T10:56:00.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:00.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:56:00.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 1 -- 172.21.15.5:0/1522070182 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c0aab80 con 0x7f779c0af6c0 2024-03-29T10:56:00.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:00.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 1 -- 172.21.15.5:0/1522070182 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7794001670 con 0x7f779c0af6c0 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.751+0000 7f779b7fe640 10 monclient: dump: 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:00.736 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.752+0000 7f779b7fe640 1 -- 172.21.15.5:0/1522070182 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7794001840 con 0x7f779c0af6c0 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.752+0000 7f779b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.752+0000 7f779b7fe640 1 -- 172.21.15.5:0/1522070182 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7794001c50 con 0x7f779c0af6c0 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.752+0000 7f779b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.752+0000 7f779b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.752+0000 7f779b7fe640 10 monclient: dump: 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:00.737 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:00.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.752+0000 7f77a30d9640 5 monclient: authenticate success, global_id 4583 2024-03-29T10:56:00.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.752+0000 7f77a30d9640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:00.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.752+0000 7f77a30d9640 4 set_mon_vals no callback set 2024-03-29T10:56:00.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.753+0000 7f77a30d9640 10 monclient: shutdown 2024-03-29T10:56:00.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.753+0000 7f77a30d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:00.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.753+0000 7f77a30d9640 1 -- 172.21.15.5:0/1522070182 >> v1:172.21.15.5:6790/0 conn(0x7f779c0af6c0 legacy=0x7f779c152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:00.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.753+0000 7f77a30d9640 1 -- 172.21.15.5:0/1522070182 shutdown_connections 2024-03-29T10:56:00.738 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.753+0000 7f77a30d9640 1 -- 172.21.15.5:0/1522070182 >> 172.21.15.5:0/1522070182 conn(0x7f779c0b35e0 msgr2=0x7f779c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.757+0000 7f77a30d9640 1 -- 172.21.15.5:0/1522070182 shutdown_connections 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.757+0000 7f77a30d9640 1 -- 172.21.15.5:0/1522070182 wait complete. 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.758+0000 7f77a30d9640 10 monclient: build_initial_monmap 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.758+0000 7f77a30d9640 10 monclient: monmap: 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:56:00.739 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:56:00.740 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:00.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.758+0000 7f77a30d9640 1 librados: starting msgr at 2024-03-29T10:56:00.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.758+0000 7f77a30d9640 1 librados: starting objecter 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.758+0000 7f77a30d9640 1 Processor -- start 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.758+0000 7f77a30d9640 1 -- start start 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.758+0000 7f77a30d9640 1 librados: setting wanted keys 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.758+0000 7f77a30d9640 1 librados: calling monclient init 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.758+0000 7f77a30d9640 10 monclient: init 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.759+0000 7f77a30d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.759+0000 7f77a30d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.759+0000 7f77a30d9640 10 monclient(hunting): picked mon.noname-b con 0x7f779c0ad6c0 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.760+0000 7f77a30d9640 10 monclient(hunting): picked mon.noname-c con 0x7f779c0af6c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.760+0000 7f77a30d9640 10 monclient(hunting): picked mon.noname-a con 0x7f779c14fdc0 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.760+0000 7f77a30d9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c1f00d0 con 0x7f779c14fdc0 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.760+0000 7f77a30d9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c072630 con 0x7f779c0ad6c0 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.760+0000 7f77a30d9640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c145ca0 con 0x7f779c0af6c0 2024-03-29T10:56:00.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.760+0000 7f77a30d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:00.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.760+0000 7f77a30d9640 10 monclient(hunting): authenticate will time out at 1374.001167s 2024-03-29T10:56:00.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f779bfff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f779c0af6c0 0x7f779c14b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:50596/0 (socket says 172.21.15.5:50596) 2024-03-29T10:56:00.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f779bfff640 1 -- 172.21.15.5:0/1540840228 learned_addr learned my addr 172.21.15.5:0/1540840228 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:00.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 client.?.objecter ms_handle_connect 0x7f779c0af6c0 2024-03-29T10:56:00.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:00.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 client.?.objecter ms_handle_connect 0x7f779c14fdc0 2024-03-29T10:56:00.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:00.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044840114 0 0) 0x7f779c145ca0 con 0x7f779c0af6c0 2024-03-29T10:56:00.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:00.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:00.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:00.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:56:00.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7788004eb0 con 0x7f779c0af6c0 2024-03-29T10:56:00.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792308226 0 0) 0x7f779c1f00d0 con 0x7f779c14fdc0 2024-03-29T10:56:00.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:00.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:00.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:00.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:56:00.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f779c145ca0 con 0x7f779c14fdc0 2024-03-29T10:56:00.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 client.?.objecter ms_handle_connect 0x7f779c0ad6c0 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.763+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1837312022 0 0) 0x7f7788004eb0 con 0x7f779c0af6c0 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f779c1f00d0 con 0x7f779c0af6c0 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f77940020d0 con 0x7f779c0af6c0 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): dump: 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:00.748 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:00.749 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:00.749 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:00.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:00.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743591467 0 0) 0x7f779c072630 con 0x7f779c0ad6c0 2024-03-29T10:56:00.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:00.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:00.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:00.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:56:00.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7788004eb0 con 0x7f779c0ad6c0 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1020890953 0 0) 0x7f779c145ca0 con 0x7f779c14fdc0 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f779c072630 con 0x7f779c14fdc0 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f778c001710 con 0x7f779c14fdc0 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient(hunting): dump: 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:00.750 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2876205371 0 0) 0x7f779c1f00d0 con 0x7f779c0af6c0 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 >> v1:172.21.15.177:6789/0 conn(0x7f779c0ad6c0 legacy=0x7f779c1f0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 >> v1:172.21.15.5:6789/0 conn(0x7f779c14fdc0 legacy=0x7f779c1f8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 monclient: found mon.c 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c1efcd0 con 0x7f779c0af6c0 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:00.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77a30d9640 5 monclient: authenticate success, global_id 4586 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77a30d9640 10 monclient: _renew_subs 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77a30d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77a30d9640 1 -- 172.21.15.5:0/1540840228 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779c14c430 con 0x7f779c0af6c0 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77a30d9640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77a30d9640 10 monclient: _renew_subs 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77a30d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77a30d9640 1 -- 172.21.15.5:0/1540840228 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f779c152120 con 0x7f779c0af6c0 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77a30d9640 1 librados: init done 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7794004fb0 con 0x7f779c0af6c0 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f77940052c0 con 0x7f779c0af6c0 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f77997fa640 10 monclient: dump: 2024-03-29T10:56:00.752 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:00.753 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:00.753 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:00.753 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:00.753 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:00.753 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:00.753 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:00.753 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:00.753 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:00.753 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:00.753 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.764+0000 7f779affd640 4 set_mon_vals no callback set 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.766+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f779401b290 con 0x7f779c0af6c0 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.766+0000 7f77a30d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.767+0000 7f77a30d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.767+0000 7f77a30d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.767+0000 7f77a30d9640 1 -- 172.21.15.5:0/1540840228 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777c005080 con 0x7f779c0af6c0 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.767+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f7794084fe0 con 0x7f779c0af6c0 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.767+0000 7f77997fa640 10 client.4586.objecter ms_dispatch 0x7f779c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.767+0000 7f77997fa640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.767+0000 7f77997fa640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.767+0000 7f77997fa640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.767+0000 7f77a0e75640 10 monclient: get_auth_request con 0x7f778806e000 auth_method 0 2024-03-29T10:56:00.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.767+0000 7f77997fa640 10 client.4586.objecter ms_handle_connect 0x7f778806e000 2024-03-29T10:56:00.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.777+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7794050940 con 0x7f779c0af6c0 2024-03-29T10:56:00.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.777+0000 7f77997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:00.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:00.777+0000 7f77997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:01.091 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.110+0000 7f77a30d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:01.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.110+0000 7f77a30d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:01.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.110+0000 7f77a30d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:01.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.111+0000 7f77a30d9640 1 -- 172.21.15.5:0/1540840228 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f777c005330 con 0x7f779c0af6c0 2024-03-29T10:56:01.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.112+0000 7f77997fa640 1 -- 172.21.15.5:0/1540840228 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f7794052870 con 0x7f779c0af6c0 2024-03-29T10:56:01.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.112+0000 7f77997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:01.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.112+0000 7f77997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:01.094 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:56:01.094 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:01.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.116+0000 7f77a30d9640 10 librados: watch_flush enter 2024-03-29T10:56:01.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.116+0000 7f77a30d9640 10 librados: watch_flush exit 2024-03-29T10:56:01.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.116+0000 7f77a30d9640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T10:56:01.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.116+0000 7f77a30d9640 10 client.4586.objecter successfully canceled tick 2024-03-29T10:56:01.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.116+0000 7f77a30d9640 1 -- 172.21.15.5:0/1540840228 >> v1:172.21.15.5:6816/2847500151 conn(0x7f778806e000 legacy=0x7f77880704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:01.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.116+0000 7f77a30d9640 10 monclient: shutdown 2024-03-29T10:56:01.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.116+0000 7f77a30d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:01.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.116+0000 7f77a30d9640 1 -- 172.21.15.5:0/1540840228 >> v1:172.21.15.5:6790/0 conn(0x7f779c0af6c0 legacy=0x7f779c14b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:01.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.117+0000 7f77a30d9640 1 -- 172.21.15.5:0/1540840228 shutdown_connections 2024-03-29T10:56:01.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.117+0000 7f77a30d9640 1 -- 172.21.15.5:0/1540840228 >> 172.21.15.5:0/1540840228 conn(0x7f779c0b35e0 msgr2=0x7f779c0a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:01.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.119+0000 7f77a30d9640 1 -- 172.21.15.5:0/1540840228 shutdown_connections 2024-03-29T10:56:01.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.119+0000 7f77a30d9640 1 -- 172.21.15.5:0/1540840228 wait complete. 2024-03-29T10:56:01.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:01.119+0000 7f77a30d9640 1 librados: shutdown 2024-03-29T10:56:01.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:07.922 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:08.072 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.091+0000 7fda4d40e640 10 monclient: get_monmap_and_config 2024-03-29T10:56:08.072 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.091+0000 7fda4d40e640 10 monclient: build_initial_monmap 2024-03-29T10:56:08.072 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.091+0000 7fda4d40e640 10 monclient: monmap: 2024-03-29T10:56:08.072 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:08.073 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:08.073 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:56:08.092561+0000 2024-03-29T10:56:08.073 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:56:08.092561+0000 2024-03-29T10:56:08.073 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:08.073 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:08.073 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:08.073 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:56:08.073 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:56:08.073 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:08.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.092+0000 7fda4d40e640 1 Processor -- start 2024-03-29T10:56:08.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.092+0000 7fda4d40e640 1 -- start start 2024-03-29T10:56:08.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.092+0000 7fda4d40e640 10 monclient: init 2024-03-29T10:56:08.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.093+0000 7fda4d40e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:08.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.093+0000 7fda4d40e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:08.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.093+0000 7fda4d40e640 10 monclient(hunting): picked mon.noname-a con 0x7fda480b1f60 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:08.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.093+0000 7fda4d40e640 10 monclient(hunting): picked mon.noname-b con 0x7fda48071eb0 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:08.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.093+0000 7fda4d40e640 10 monclient(hunting): picked mon.noname-c con 0x7fda48143d10 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:08.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.093+0000 7fda4d40e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4806ef80 con 0x7fda480b1f60 2024-03-29T10:56:08.074 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.093+0000 7fda4d40e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4806f260 con 0x7fda48143d10 2024-03-29T10:56:08.075 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.093+0000 7fda4d40e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda480b40c0 con 0x7fda48071eb0 2024-03-29T10:56:08.075 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.093+0000 7fda4d40e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:08.075 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.093+0000 7fda4d40e640 10 monclient(hunting): authenticate will time out at 1381.334554s 2024-03-29T10:56:08.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.094+0000 7fda467fc640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fda48071eb0 0x7fda48146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:59486/0 (socket says 172.21.15.5:59486) 2024-03-29T10:56:08.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.094+0000 7fda467fc640 1 -- 172.21.15.5:0/3470022318 learned_addr learned my addr 172.21.15.5:0/3470022318 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:08.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.094+0000 7fda45ffb640 1 -- 172.21.15.5:0/3470022318 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43045436 0 0) 0x7fda480b40c0 con 0x7fda48071eb0 2024-03-29T10:56:08.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.094+0000 7fda45ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:08.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.094+0000 7fda45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:08.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.094+0000 7fda45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:08.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.094+0000 7fda45ffb640 10 monclient(hunting): my global_id is 4589 2024-03-29T10:56:08.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.094+0000 7fda45ffb640 1 -- 172.21.15.5:0/3470022318 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda30003520 con 0x7fda48071eb0 2024-03-29T10:56:08.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.094+0000 7fda45ffb640 1 -- 172.21.15.5:0/3470022318 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3444594232 0 0) 0x7fda30003520 con 0x7fda48071eb0 2024-03-29T10:56:08.081 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.095+0000 7fda45ffb640 1 -- 172.21.15.5:0/3470022318 >> v1:172.21.15.177:6789/0 conn(0x7fda48143d10 legacy=0x7fda48146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 1 -- 172.21.15.5:0/3470022318 >> v1:172.21.15.5:6789/0 conn(0x7fda480b1f60 legacy=0x7fda48143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 1 monclient: found mon.noname-b 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 1 -- 172.21.15.5:0/3470022318 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda48155d70 con 0x7fda48071eb0 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 10 monclient: _finish_auth 0 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda4d40e640 5 monclient: authenticate success, global_id 4589 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda4d40e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 1 -- 172.21.15.5:0/3470022318 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fda38001690 con 0x7fda48071eb0 2024-03-29T10:56:08.082 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda45ffb640 10 monclient: dump: 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.097+0000 7fda4d40e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.098+0000 7fda45ffb640 1 -- 172.21.15.5:0/3470022318 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda38001860 con 0x7fda48071eb0 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.098+0000 7fda45ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:08.083 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.098+0000 7fda45ffb640 1 -- 172.21.15.5:0/3470022318 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fda38001cb0 con 0x7fda48071eb0 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.098+0000 7fda45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.098+0000 7fda45ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.098+0000 7fda45ffb640 10 monclient: dump: 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.099+0000 7fda4d40e640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.099+0000 7fda4d40e640 4 set_mon_vals no callback set 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.100+0000 7fda4d40e640 10 monclient: shutdown 2024-03-29T10:56:08.084 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.100+0000 7fda4d40e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:08.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.100+0000 7fda4d40e640 1 -- 172.21.15.5:0/3470022318 >> v1:172.21.15.5:6790/0 conn(0x7fda48071eb0 legacy=0x7fda48146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:08.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.100+0000 7fda4d40e640 1 -- 172.21.15.5:0/3470022318 shutdown_connections 2024-03-29T10:56:08.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.100+0000 7fda4d40e640 1 -- 172.21.15.5:0/3470022318 >> 172.21.15.5:0/3470022318 conn(0x7fda48066a00 msgr2=0x7fda48066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:08.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.103+0000 7fda4d40e640 1 -- 172.21.15.5:0/3470022318 shutdown_connections 2024-03-29T10:56:08.085 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.104+0000 7fda4d40e640 1 -- 172.21.15.5:0/3470022318 wait complete. 2024-03-29T10:56:08.086 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.104+0000 7fda4d40e640 10 monclient: build_initial_monmap 2024-03-29T10:56:08.086 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.105+0000 7fda4d40e640 10 monclient: monmap: 2024-03-29T10:56:08.086 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:08.086 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.105+0000 7fda4d40e640 1 librados: starting msgr at 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.105+0000 7fda4d40e640 1 librados: starting objecter 2024-03-29T10:56:08.087 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.105+0000 7fda4d40e640 1 Processor -- start 2024-03-29T10:56:08.091 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 1 -- start start 2024-03-29T10:56:08.091 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 1 librados: setting wanted keys 2024-03-29T10:56:08.091 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 1 librados: calling monclient init 2024-03-29T10:56:08.091 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 10 monclient: init 2024-03-29T10:56:08.091 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:08.091 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 10 monclient(hunting): picked mon.noname-a con 0x7fda480b1f60 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 10 monclient(hunting): picked mon.noname-c con 0x7fda48143d10 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 10 monclient(hunting): picked mon.noname-b con 0x7fda481492a0 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda48149dc0 con 0x7fda480b1f60 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4814b440 con 0x7fda481492a0 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4814a250 con 0x7fda48143d10 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.106+0000 7fda4d40e640 10 monclient(hunting): authenticate will time out at 1381.347919s 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.109+0000 7fda467fc640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fda48143d10 0x7fda4813afa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:33494/0 (socket says 172.21.15.5:33494) 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.109+0000 7fda467fc640 1 -- 172.21.15.5:0/3707039782 learned_addr learned my addr 172.21.15.5:0/3707039782 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.109+0000 7fda2f7fe640 10 client.?.objecter ms_handle_connect 0x7fda48143d10 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.109+0000 7fda2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.109+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117794743 0 0) 0x7fda4814a250 con 0x7fda48143d10 2024-03-29T10:56:08.092 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.109+0000 7fda2f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.109+0000 7fda2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.109+0000 7fda2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.109+0000 7fda2f7fe640 10 monclient(hunting): my global_id is 4592 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.109+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda34004eb0 con 0x7fda48143d10 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.110+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2930052543 0 0) 0x7fda34004eb0 con 0x7fda48143d10 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.110+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda4814a250 con 0x7fda48143d10 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.110+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fda38001b70 con 0x7fda48143d10 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.110+0000 7fda2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.110+0000 7fda2f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.110+0000 7fda2f7fe640 10 monclient(hunting): dump: 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:08.093 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:08.094 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:08.094 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:08.094 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:08.094 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:08.094 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:08.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.110+0000 7fda2f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:08.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.110+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1208780313 0 0) 0x7fda4814a250 con 0x7fda48143d10 2024-03-29T10:56:08.094 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.110+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 >> v1:172.21.15.177:6789/0 conn(0x7fda481492a0 legacy=0x7fda481e6eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:08.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 >> v1:172.21.15.5:6789/0 conn(0x7fda480b1f60 legacy=0x7fda48147380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:08.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda2f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:08.095 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda2f7fe640 1 monclient: found mon.c 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda2f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda48147a90 con 0x7fda48143d10 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda2f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda2f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda4d40e640 5 monclient: authenticate success, global_id 4592 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda4d40e640 10 monclient: _renew_subs 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda4d40e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda4d40e640 1 -- 172.21.15.5:0/3707039782 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda48147d40 con 0x7fda48143d10 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda4d40e640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda4d40e640 10 monclient: _renew_subs 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda4d40e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:08.096 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda4d40e640 1 -- 172.21.15.5:0/3707039782 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda48154e40 con 0x7fda48143d10 2024-03-29T10:56:08.097 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.113+0000 7fda4d40e640 1 librados: init done 2024-03-29T10:56:08.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.114+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda38004bf0 con 0x7fda48143d10 2024-03-29T10:56:08.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.115+0000 7fda2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:08.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.115+0000 7fda457fa640 4 set_mon_vals no callback set 2024-03-29T10:56:08.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.115+0000 7fda4d40e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:08.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.116+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fda38005000 con 0x7fda48143d10 2024-03-29T10:56:08.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.116+0000 7fda2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:08.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.116+0000 7fda2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:08.098 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.116+0000 7fda2f7fe640 10 monclient: dump: 2024-03-29T10:56:08.098 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.116+0000 7fda4d40e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.116+0000 7fda4d40e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.116+0000 7fda4d40e640 1 -- 172.21.15.5:0/3707039782 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda28005080 con 0x7fda48143d10 2024-03-29T10:56:08.099 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.117+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fda3801afd0 con 0x7fda48143d10 2024-03-29T10:56:08.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.118+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fda38084c60 con 0x7fda48143d10 2024-03-29T10:56:08.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.118+0000 7fda2f7fe640 10 client.4592.objecter ms_dispatch 0x7fda48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:08.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.118+0000 7fda2f7fe640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:08.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.118+0000 7fda2f7fe640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:08.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.118+0000 7fda2f7fe640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T10:56:08.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.118+0000 7fda46ffd640 10 monclient: get_auth_request con 0x7fda3406dc10 auth_method 0 2024-03-29T10:56:08.100 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.119+0000 7fda2f7fe640 10 client.4592.objecter ms_handle_connect 0x7fda3406dc10 2024-03-29T10:56:08.106 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.125+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda38050540 con 0x7fda48143d10 2024-03-29T10:56:08.106 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.125+0000 7fda2f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:08.106 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.126+0000 7fda2f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:08.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.459+0000 7fda4d40e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:08.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.459+0000 7fda4d40e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:08.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.459+0000 7fda4d40e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:08.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.459+0000 7fda4d40e640 1 -- 172.21.15.5:0/3707039782 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda28005330 con 0x7fda48143d10 2024-03-29T10:56:08.442 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.461+0000 7fda2f7fe640 1 -- 172.21.15.5:0/3707039782 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fda38052470 con 0x7fda48143d10 2024-03-29T10:56:08.442 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.461+0000 7fda2f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:08.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.461+0000 7fda2f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:08.443 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:56:08.443 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:08.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.465+0000 7fda4d40e640 10 librados: watch_flush enter 2024-03-29T10:56:08.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.465+0000 7fda4d40e640 10 librados: watch_flush exit 2024-03-29T10:56:08.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.465+0000 7fda4d40e640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T10:56:08.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.465+0000 7fda4d40e640 10 client.4592.objecter successfully canceled tick 2024-03-29T10:56:08.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.465+0000 7fda4d40e640 1 -- 172.21.15.5:0/3707039782 >> v1:172.21.15.5:6816/2847500151 conn(0x7fda3406dc10 legacy=0x7fda340700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:08.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.465+0000 7fda4d40e640 10 monclient: shutdown 2024-03-29T10:56:08.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.465+0000 7fda4d40e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:08.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.465+0000 7fda4d40e640 1 -- 172.21.15.5:0/3707039782 >> v1:172.21.15.5:6790/0 conn(0x7fda48143d10 legacy=0x7fda4813afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:08.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.465+0000 7fda4d40e640 1 -- 172.21.15.5:0/3707039782 shutdown_connections 2024-03-29T10:56:08.451 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.465+0000 7fda4d40e640 1 -- 172.21.15.5:0/3707039782 >> 172.21.15.5:0/3707039782 conn(0x7fda48066a00 msgr2=0x7fda480b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:08.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.471+0000 7fda4d40e640 1 -- 172.21.15.5:0/3707039782 shutdown_connections 2024-03-29T10:56:08.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.471+0000 7fda4d40e640 1 -- 172.21.15.5:0/3707039782 wait complete. 2024-03-29T10:56:08.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:08.471+0000 7fda4d40e640 1 librados: shutdown 2024-03-29T10:56:08.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:15.278 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:15.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.448+0000 7f889d1a4640 10 monclient: get_monmap_and_config 2024-03-29T10:56:15.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.448+0000 7f889d1a4640 10 monclient: build_initial_monmap 2024-03-29T10:56:15.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.448+0000 7f889d1a4640 10 monclient: monmap: 2024-03-29T10:56:15.429 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:15.429 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:15.429 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:56:15.449474+0000 2024-03-29T10:56:15.429 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:56:15.449474+0000 2024-03-29T10:56:15.429 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:15.430 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:15.430 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:15.430 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:56:15.430 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:56:15.430 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:15.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.449+0000 7f889d1a4640 1 Processor -- start 2024-03-29T10:56:15.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.449+0000 7f889d1a4640 1 -- start start 2024-03-29T10:56:15.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.449+0000 7f889d1a4640 10 monclient: init 2024-03-29T10:56:15.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.450+0000 7f889d1a4640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:15.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.450+0000 7f889d1a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:15.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.450+0000 7f889d1a4640 10 monclient(hunting): picked mon.noname-b con 0x7f88900389e0 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:15.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.450+0000 7f889d1a4640 10 monclient(hunting): picked mon.noname-a con 0x7f8890038dc0 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:15.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.450+0000 7f889d1a4640 10 monclient(hunting): picked mon.noname-c con 0x7f8890030660 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:15.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.450+0000 7f889d1a4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900c54c0 con 0x7f8890038dc0 2024-03-29T10:56:15.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.450+0000 7f889d1a4640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8890036430 con 0x7f8890030660 2024-03-29T10:56:15.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.450+0000 7f889d1a4640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8890039840 con 0x7f88900389e0 2024-03-29T10:56:15.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.450+0000 7f889d1a4640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:15.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.450+0000 7f889d1a4640 10 monclient(hunting): authenticate will time out at 1388.691808s 2024-03-29T10:56:15.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.452+0000 7f8897fff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f88900389e0 0x7f8890033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:38854/0 (socket says 172.21.15.5:38854) 2024-03-29T10:56:15.433 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.452+0000 7f8897fff640 1 -- 172.21.15.5:0/3638652515 learned_addr learned my addr 172.21.15.5:0/3638652515 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:15.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.454+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177136963 0 0) 0x7f8890039840 con 0x7f88900389e0 2024-03-29T10:56:15.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.454+0000 7f8896ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:15.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.454+0000 7f8896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:15.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.454+0000 7f8896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:15.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.454+0000 7f8896ffd640 10 monclient(hunting): my global_id is 4595 2024-03-29T10:56:15.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.454+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8880003520 con 0x7f88900389e0 2024-03-29T10:56:15.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.455+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554120186 0 0) 0x7f88900c54c0 con 0x7f8890038dc0 2024-03-29T10:56:15.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.455+0000 7f8896ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:15.436 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.455+0000 7f8896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:15.436 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.455+0000 7f8896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:15.436 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.455+0000 7f8896ffd640 10 monclient(hunting): my global_id is 4494 2024-03-29T10:56:15.436 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.455+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8890039840 con 0x7f8890038dc0 2024-03-29T10:56:15.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2926849589 0 0) 0x7f8880003520 con 0x7f88900389e0 2024-03-29T10:56:15.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 >> v1:172.21.15.177:6789/0 conn(0x7f8890030660 legacy=0x7f88900d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:15.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 >> v1:172.21.15.5:6789/0 conn(0x7f8890038dc0 legacy=0x7f88900cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:15.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:15.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 1 monclient: found mon.noname-b 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8890035690 con 0x7f88900389e0 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f889804f730 con 0x7f88900389e0 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.458+0000 7f8896ffd640 10 monclient: dump: 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:15.440 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:15.441 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:15.441 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:15.441 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:15.441 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:15.441 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:15.441 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:15.441 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.460+0000 7f889d1a4640 5 monclient: authenticate success, global_id 4595 2024-03-29T10:56:15.441 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.460+0000 7f889d1a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:15.442 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.461+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88980008d0 con 0x7f88900389e0 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.461+0000 7f8896ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.461+0000 7f8896ffd640 1 -- 172.21.15.5:0/3638652515 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f88980758b0 con 0x7f88900389e0 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.461+0000 7f8896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.461+0000 7f8896ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.461+0000 7f8896ffd640 10 monclient: dump: 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:15.443 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:15.444 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:15.444 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.462+0000 7f889d1a4640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:15.444 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.462+0000 7f889d1a4640 4 set_mon_vals no callback set 2024-03-29T10:56:15.445 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.463+0000 7f889d1a4640 10 monclient: shutdown 2024-03-29T10:56:15.445 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.463+0000 7f889d1a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:15.445 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.463+0000 7f889d1a4640 1 -- 172.21.15.5:0/3638652515 >> v1:172.21.15.5:6790/0 conn(0x7f88900389e0 legacy=0x7f8890033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:15.445 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.463+0000 7f889d1a4640 1 -- 172.21.15.5:0/3638652515 shutdown_connections 2024-03-29T10:56:15.445 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.463+0000 7f889d1a4640 1 -- 172.21.15.5:0/3638652515 >> 172.21.15.5:0/3638652515 conn(0x7f8890005c00 msgr2=0x7f8890006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:15.445 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.464+0000 7f889d1a4640 1 -- 172.21.15.5:0/3638652515 shutdown_connections 2024-03-29T10:56:15.447 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.466+0000 7f889d1a4640 1 -- 172.21.15.5:0/3638652515 wait complete. 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.467+0000 7f889d1a4640 10 monclient: build_initial_monmap 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.467+0000 7f889d1a4640 10 monclient: monmap: 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:56:15.448 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:15.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.467+0000 7f889d1a4640 1 librados: starting msgr at 2024-03-29T10:56:15.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.467+0000 7f889d1a4640 1 librados: starting objecter 2024-03-29T10:56:15.449 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.468+0000 7f889d1a4640 1 Processor -- start 2024-03-29T10:56:15.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.468+0000 7f889d1a4640 1 -- start start 2024-03-29T10:56:15.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.468+0000 7f889d1a4640 1 librados: setting wanted keys 2024-03-29T10:56:15.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.468+0000 7f889d1a4640 1 librados: calling monclient init 2024-03-29T10:56:15.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.468+0000 7f889d1a4640 10 monclient: init 2024-03-29T10:56:15.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.469+0000 7f889d1a4640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:15.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.469+0000 7f889d1a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:15.450 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.469+0000 7f889d1a4640 10 monclient(hunting): picked mon.noname-c con 0x7f8890030660 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:15.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.469+0000 7f889d1a4640 10 monclient(hunting): picked mon.noname-b con 0x7f889002f1c0 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:15.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.471+0000 7f889d1a4640 10 monclient(hunting): picked mon.noname-a con 0x7f88900c7880 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:15.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.471+0000 7f889d1a4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900c7fe0 con 0x7f88900c7880 2024-03-29T10:56:15.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.471+0000 7f889d1a4640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900c93b0 con 0x7f889002f1c0 2024-03-29T10:56:15.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.471+0000 7f889d1a4640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900c7cd0 con 0x7f8890030660 2024-03-29T10:56:15.452 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.471+0000 7f889d1a4640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:15.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.471+0000 7f889d1a4640 10 monclient(hunting): authenticate will time out at 1388.712329s 2024-03-29T10:56:15.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.470+0000 7f8897fff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f8890030660 0x7f889015c6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:42098/0 (socket says 172.21.15.5:42098) 2024-03-29T10:56:15.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.471+0000 7f8897fff640 1 -- 172.21.15.5:0/1472158045 learned_addr learned my addr 172.21.15.5:0/1472158045 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:15.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.472+0000 7f8894ff9640 10 client.?.objecter ms_handle_connect 0x7f8890030660 2024-03-29T10:56:15.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.472+0000 7f8894ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:15.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.472+0000 7f8894ff9640 10 client.?.objecter ms_handle_connect 0x7f889002f1c0 2024-03-29T10:56:15.453 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.472+0000 7f8894ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:15.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927256330 0 0) 0x7f88900c7cd0 con 0x7f8890030660 2024-03-29T10:56:15.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:15.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:15.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:15.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:56:15.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8884004eb0 con 0x7f8890030660 2024-03-29T10:56:15.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906314707 0 0) 0x7f88900c93b0 con 0x7f889002f1c0 2024-03-29T10:56:15.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:15.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:15.454 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:15.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:56:15.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.473+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88900c7cd0 con 0x7f889002f1c0 2024-03-29T10:56:15.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.474+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4155547728 0 0) 0x7f8884004eb0 con 0x7f8890030660 2024-03-29T10:56:15.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.474+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88900c93b0 con 0x7f8890030660 2024-03-29T10:56:15.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.474+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f889804f400 con 0x7f8890030660 2024-03-29T10:56:15.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.474+0000 7f8894ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:15.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.474+0000 7f8894ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:15.455 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.474+0000 7f8894ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:15.455 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:15.455 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:15.456 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:15.456 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:15.456 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:15.456 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:15.456 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:15.456 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:15.456 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:15.456 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:15.456 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.474+0000 7f8894ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:15.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.475+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2652083421 0 0) 0x7f88900c93b0 con 0x7f8890030660 2024-03-29T10:56:15.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.475+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 >> v1:172.21.15.177:6789/0 conn(0x7f889002f1c0 legacy=0x7f889015ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:15.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.475+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 >> v1:172.21.15.5:6789/0 conn(0x7f88900c7880 legacy=0x7f8890167280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:15.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.475+0000 7f8894ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:15.458 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.475+0000 7f8894ff9640 1 monclient: found mon.c 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.475+0000 7f8894ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.475+0000 7f8894ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.475+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900c74a0 con 0x7f8890030660 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f8894ff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f8894ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f8894ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f889d1a4640 5 monclient: authenticate success, global_id 4598 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f889d1a4640 10 monclient: _renew_subs 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f889d1a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f889d1a4640 1 -- 172.21.15.5:0/1472158045 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88901689a0 con 0x7f8890030660 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f889d1a4640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f889d1a4640 10 monclient: _renew_subs 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f889d1a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f889d1a4640 1 -- 172.21.15.5:0/1472158045 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8890168d10 con 0x7f8890030660 2024-03-29T10:56:15.459 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f889d1a4640 1 librados: init done 2024-03-29T10:56:15.460 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.476+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f889804f730 con 0x7f8890030660 2024-03-29T10:56:15.460 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.477+0000 7f8894ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:15.460 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.477+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f8898067d90 con 0x7f8890030660 2024-03-29T10:56:15.460 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.477+0000 7f8894ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:15.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.477+0000 7f8894ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:15.462 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.477+0000 7f8894ff9640 10 monclient: dump: 2024-03-29T10:56:15.462 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:15.462 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:15.462 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:15.462 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:15.462 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:15.462 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:15.462 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:15.462 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:15.463 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:15.463 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:15.463 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.477+0000 7f8895ffb640 4 set_mon_vals no callback set 2024-03-29T10:56:15.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.481+0000 7f889d1a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:15.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.481+0000 7f889d1a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:15.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.481+0000 7f889d1a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:15.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.481+0000 7f889d1a4640 1 -- 172.21.15.5:0/1472158045 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8890007b90 con 0x7f8890030660 2024-03-29T10:56:15.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.482+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f8898067ef0 con 0x7f8890030660 2024-03-29T10:56:15.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.483+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f88980ff4c0 con 0x7f8890030660 2024-03-29T10:56:15.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.483+0000 7f8894ff9640 10 client.4598.objecter ms_dispatch 0x7f8898000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:15.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.483+0000 7f8894ff9640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:15.464 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.483+0000 7f8894ff9640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:15.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.483+0000 7f88977fe640 10 monclient: get_auth_request con 0x7f888406dba0 auth_method 0 2024-03-29T10:56:15.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.483+0000 7f8894ff9640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T10:56:15.465 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.483+0000 7f8894ff9640 10 client.4598.objecter ms_handle_connect 0x7f888406dba0 2024-03-29T10:56:15.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.498+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88980cada0 con 0x7f8890030660 2024-03-29T10:56:15.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.498+0000 7f8894ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:15.478 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.498+0000 7f8894ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:15.891 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.910+0000 7f889d1a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:15.891 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.910+0000 7f889d1a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:15.891 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.910+0000 7f889d1a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:15.891 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.910+0000 7f889d1a4640 1 -- 172.21.15.5:0/1472158045 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8890005120 con 0x7f8890030660 2024-03-29T10:56:15.892 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.911+0000 7f8894ff9640 1 -- 172.21.15.5:0/1472158045 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f88980cccd0 con 0x7f8890030660 2024-03-29T10:56:15.892 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.911+0000 7f8894ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:15.892 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.911+0000 7f8894ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:15.893 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:56:15.893 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:15.898 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.916+0000 7f889d1a4640 10 librados: watch_flush enter 2024-03-29T10:56:15.898 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.916+0000 7f889d1a4640 10 librados: watch_flush exit 2024-03-29T10:56:15.898 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.916+0000 7f889d1a4640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T10:56:15.898 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.916+0000 7f889d1a4640 10 client.4598.objecter successfully canceled tick 2024-03-29T10:56:15.898 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.916+0000 7f889d1a4640 1 -- 172.21.15.5:0/1472158045 >> v1:172.21.15.5:6816/2847500151 conn(0x7f888406dba0 legacy=0x7f8884070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:15.898 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.917+0000 7f889d1a4640 10 monclient: shutdown 2024-03-29T10:56:15.898 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.917+0000 7f889d1a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:15.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.917+0000 7f889d1a4640 1 -- 172.21.15.5:0/1472158045 >> v1:172.21.15.5:6790/0 conn(0x7f8890030660 legacy=0x7f889015c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:15.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.917+0000 7f889d1a4640 1 -- 172.21.15.5:0/1472158045 shutdown_connections 2024-03-29T10:56:15.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.917+0000 7f889d1a4640 1 -- 172.21.15.5:0/1472158045 >> 172.21.15.5:0/1472158045 conn(0x7f8890005c00 msgr2=0x7f8890034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:15.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.917+0000 7f889d1a4640 1 -- 172.21.15.5:0/1472158045 shutdown_connections 2024-03-29T10:56:15.899 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.919+0000 7f889d1a4640 1 -- 172.21.15.5:0/1472158045 wait complete. 2024-03-29T10:56:15.900 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:15.919+0000 7f889d1a4640 1 librados: shutdown 2024-03-29T10:56:15.920 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:22.727 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:22.867 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.886+0000 7fe9ea649640 10 monclient: get_monmap_and_config 2024-03-29T10:56:22.867 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.886+0000 7fe9ea649640 10 monclient: build_initial_monmap 2024-03-29T10:56:22.867 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.886+0000 7fe9ea649640 10 monclient: monmap: 2024-03-29T10:56:22.867 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:22.868 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:22.868 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:56:22.887592+0000 2024-03-29T10:56:22.868 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:56:22.887592+0000 2024-03-29T10:56:22.868 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:22.868 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:22.868 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:22.868 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:56:22.868 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:56:22.868 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:22.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.887+0000 7fe9ea649640 1 Processor -- start 2024-03-29T10:56:22.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.887+0000 7fe9ea649640 1 -- start start 2024-03-29T10:56:22.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.887+0000 7fe9ea649640 10 monclient: init 2024-03-29T10:56:22.869 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.888+0000 7fe9ea649640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:22.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.888+0000 7fe9ea649640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:22.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.888+0000 7fe9ea649640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e40b0770 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:22.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.888+0000 7fe9ea649640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e414a700 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:22.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.888+0000 7fe9ea649640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e40ac5a0 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:22.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.888+0000 7fe9ea649640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e40aba70 con 0x7fe9e40b0770 2024-03-29T10:56:22.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.888+0000 7fe9ea649640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e40aecb0 con 0x7fe9e414a700 2024-03-29T10:56:22.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.888+0000 7fe9ea649640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e40af150 con 0x7fe9e40ac5a0 2024-03-29T10:56:22.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.888+0000 7fe9ea649640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:22.870 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.888+0000 7fe9ea649640 10 monclient(hunting): authenticate will time out at 1396.129773s 2024-03-29T10:56:22.871 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.889+0000 7fe9e8be6640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fe9e40ac5a0 0x7fe9e4151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:57668/0 (socket says 172.21.15.5:57668) 2024-03-29T10:56:22.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.889+0000 7fe9e8be6640 1 -- 172.21.15.5:0/4078368694 learned_addr learned my addr 172.21.15.5:0/4078368694 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:22.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.890+0000 7fe9e2ffd640 1 -- 172.21.15.5:0/4078368694 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389539252 0 0) 0x7fe9e40af150 con 0x7fe9e40ac5a0 2024-03-29T10:56:22.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.890+0000 7fe9e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:22.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.890+0000 7fe9e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:22.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.890+0000 7fe9e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:22.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.890+0000 7fe9e2ffd640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:56:22.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.890+0000 7fe9e2ffd640 1 -- 172.21.15.5:0/4078368694 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9cc003520 con 0x7fe9e40ac5a0 2024-03-29T10:56:22.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.890+0000 7fe9e2ffd640 1 -- 172.21.15.5:0/4078368694 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2047040195 0 0) 0x7fe9cc003520 con 0x7fe9e40ac5a0 2024-03-29T10:56:22.872 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.890+0000 7fe9e2ffd640 1 -- 172.21.15.5:0/4078368694 >> v1:172.21.15.177:6789/0 conn(0x7fe9e414a700 legacy=0x7fe9e40abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 1 -- 172.21.15.5:0/4078368694 >> v1:172.21.15.5:6789/0 conn(0x7fe9e40b0770 legacy=0x7fe9e4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 1 monclient: found mon.noname-b 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 1 -- 172.21.15.5:0/4078368694 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e40ac9d0 con 0x7fe9e40ac5a0 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 1 -- 172.21.15.5:0/4078368694 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe9d4001690 con 0x7fe9e40ac5a0 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:22.873 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9e2ffd640 10 monclient: dump: 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9ea649640 5 monclient: authenticate success, global_id 4601 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.891+0000 7fe9ea649640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:22.874 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.892+0000 7fe9e2ffd640 1 -- 172.21.15.5:0/4078368694 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9d4001860 con 0x7fe9e40ac5a0 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.892+0000 7fe9e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.892+0000 7fe9e2ffd640 1 -- 172.21.15.5:0/4078368694 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe9d4001c70 con 0x7fe9e40ac5a0 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.892+0000 7fe9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.892+0000 7fe9e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.892+0000 7fe9e2ffd640 10 monclient: dump: 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:22.875 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:22.876 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:22.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.892+0000 7fe9ea649640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:22.876 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.892+0000 7fe9ea649640 4 set_mon_vals no callback set 2024-03-29T10:56:22.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.893+0000 7fe9ea649640 10 monclient: shutdown 2024-03-29T10:56:22.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.893+0000 7fe9ea649640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:22.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.893+0000 7fe9ea649640 1 -- 172.21.15.5:0/4078368694 >> v1:172.21.15.5:6790/0 conn(0x7fe9e40ac5a0 legacy=0x7fe9e4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.877 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.893+0000 7fe9ea649640 1 -- 172.21.15.5:0/4078368694 shutdown_connections 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.893+0000 7fe9ea649640 1 -- 172.21.15.5:0/4078368694 >> 172.21.15.5:0/4078368694 conn(0x7fe9e406b660 msgr2=0x7fe9e406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.894+0000 7fe9ea649640 1 -- 172.21.15.5:0/4078368694 shutdown_connections 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.894+0000 7fe9ea649640 1 -- 172.21.15.5:0/4078368694 wait complete. 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.895+0000 7fe9ea649640 10 monclient: build_initial_monmap 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.895+0000 7fe9ea649640 10 monclient: monmap: 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:22.878 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:56:22.879 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:56:22.879 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:22.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.895+0000 7fe9ea649640 1 librados: starting msgr at 2024-03-29T10:56:22.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.895+0000 7fe9ea649640 1 librados: starting objecter 2024-03-29T10:56:22.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.895+0000 7fe9ea649640 1 Processor -- start 2024-03-29T10:56:22.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.896+0000 7fe9ea649640 1 -- start start 2024-03-29T10:56:22.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.896+0000 7fe9ea649640 1 librados: setting wanted keys 2024-03-29T10:56:22.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.896+0000 7fe9ea649640 1 librados: calling monclient init 2024-03-29T10:56:22.879 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.896+0000 7fe9ea649640 10 monclient: init 2024-03-29T10:56:22.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.897+0000 7fe9ea649640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:22.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.897+0000 7fe9ea649640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:22.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.897+0000 7fe9ea649640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e40ac5a0 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:22.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9ea649640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e40b0770 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:22.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9ea649640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e414a700 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:22.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9ea649640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e41dbf10 con 0x7fe9e40b0770 2024-03-29T10:56:22.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9ea649640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e41dc1d0 con 0x7fe9e40ac5a0 2024-03-29T10:56:22.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9ea649640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e4145840 con 0x7fe9e414a700 2024-03-29T10:56:22.884 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9ea649640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9ea649640 10 monclient(hunting): authenticate will time out at 1396.139622s 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9e8be6640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fe9e414a700 0x7fe9e41deb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:58280/0 (socket says 172.21.15.5:58280) 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9e8be6640 1 -- 172.21.15.5:0/32565092 learned_addr learned my addr 172.21.15.5:0/32565092 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9e0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9e414a700 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.898+0000 7fe9e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745535141 0 0) 0x7fe9e4145840 con 0x7fe9e414a700 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 10 monclient(hunting): my global_id is 4604 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d0004eb0 con 0x7fe9e414a700 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1403121898 0 0) 0x7fe9d0004eb0 con 0x7fe9e414a700 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9e4145840 con 0x7fe9e414a700 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe9d4001b70 con 0x7fe9e414a700 2024-03-29T10:56:22.885 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.899+0000 7fe9e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:22.886 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.900+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2310432794 0 0) 0x7fe9e4145840 con 0x7fe9e414a700 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.900+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 >> v1:172.21.15.177:6789/0 conn(0x7fe9e40ac5a0 legacy=0x7fe9e41db800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 >> v1:172.21.15.5:6789/0 conn(0x7fe9e40b0770 legacy=0x7fe9e4144be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 1 monclient: found mon.c 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e41459d0 con 0x7fe9e414a700 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9e40b0770 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9ea649640 5 monclient: authenticate success, global_id 4604 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9ea649640 10 monclient: _renew_subs 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9ea649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:22.887 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9ea649640 1 -- 172.21.15.5:0/32565092 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e41452f0 con 0x7fe9e414a700 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.902+0000 7fe9ea649640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9ea649640 10 monclient: _renew_subs 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9ea649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9ea649640 1 -- 172.21.15.5:0/32565092 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e40713a0 con 0x7fe9e414a700 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9ea649640 1 librados: init done 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9d4004bf0 con 0x7fe9e414a700 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fe9d4004f00 con 0x7fe9e414a700 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9e0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9e27fc640 4 set_mon_vals no callback set 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.903+0000 7fe9e0ff9640 10 monclient: dump: 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:22.888 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:22.889 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:22.889 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:22.889 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:22.889 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:22.889 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:22.889 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.904+0000 7fe9ea649640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.905+0000 7fe9ea649640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.905+0000 7fe9ea649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.905+0000 7fe9ea649640 1 -- 172.21.15.5:0/32565092 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c4005080 con 0x7fe9e414a700 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.905+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fe9d401aed0 con 0x7fe9e414a700 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.906+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fe9d4084bb0 con 0x7fe9e414a700 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.906+0000 7fe9e0ff9640 10 client.4604.objecter ms_dispatch 0x7fe9e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.906+0000 7fe9e0ff9640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.906+0000 7fe9e0ff9640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.906+0000 7fe9e3fff640 10 monclient: get_auth_request con 0x7fe9d006dac0 auth_method 0 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.906+0000 7fe9e0ff9640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T10:56:22.890 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.907+0000 7fe9e0ff9640 10 client.4604.objecter ms_handle_connect 0x7fe9d006dac0 2024-03-29T10:56:22.894 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.912+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9d40504f0 con 0x7fe9e414a700 2024-03-29T10:56:22.894 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.913+0000 7fe9e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:22.894 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:22.913+0000 7fe9e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:23.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.250+0000 7fe9ea649640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:23.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.250+0000 7fe9ea649640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:23.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.250+0000 7fe9ea649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:23.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.250+0000 7fe9ea649640 1 -- 172.21.15.5:0/32565092 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9c4005330 con 0x7fe9e414a700 2024-03-29T10:56:23.233 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.252+0000 7fe9e0ff9640 1 -- 172.21.15.5:0/32565092 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fe9d4084260 con 0x7fe9e414a700 2024-03-29T10:56:23.233 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.252+0000 7fe9e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:23.234 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.252+0000 7fe9e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:23.234 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:56:23.234 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:23.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.257+0000 7fe9ea649640 10 librados: watch_flush enter 2024-03-29T10:56:23.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.257+0000 7fe9ea649640 10 librados: watch_flush exit 2024-03-29T10:56:23.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.257+0000 7fe9ea649640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T10:56:23.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.257+0000 7fe9ea649640 10 client.4604.objecter successfully canceled tick 2024-03-29T10:56:23.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.257+0000 7fe9ea649640 1 -- 172.21.15.5:0/32565092 >> v1:172.21.15.5:6816/2847500151 conn(0x7fe9d006dac0 legacy=0x7fe9d006ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:23.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.257+0000 7fe9ea649640 10 monclient: shutdown 2024-03-29T10:56:23.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.257+0000 7fe9ea649640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:23.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.257+0000 7fe9ea649640 1 -- 172.21.15.5:0/32565092 >> v1:172.21.15.5:6790/0 conn(0x7fe9e414a700 legacy=0x7fe9e41deb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:23.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.257+0000 7fe9ea649640 1 -- 172.21.15.5:0/32565092 shutdown_connections 2024-03-29T10:56:23.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.257+0000 7fe9ea649640 1 -- 172.21.15.5:0/32565092 >> 172.21.15.5:0/32565092 conn(0x7fe9e406b660 msgr2=0x7fe9e406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:23.239 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.258+0000 7fe9ea649640 1 -- 172.21.15.5:0/32565092 shutdown_connections 2024-03-29T10:56:23.239 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.258+0000 7fe9ea649640 1 -- 172.21.15.5:0/32565092 wait complete. 2024-03-29T10:56:23.239 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:23.259+0000 7fe9ea649640 1 librados: shutdown 2024-03-29T10:56:23.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:30.063 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.229+0000 7f2d6e7d6640 10 monclient: get_monmap_and_config 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.229+0000 7f2d6e7d6640 10 monclient: build_initial_monmap 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.229+0000 7f2d6e7d6640 10 monclient: monmap: 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:56:30.230097+0000 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:56:30.230097+0000 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:30.210 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:56:30.211 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:56:30.211 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:30.211 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.230+0000 7f2d6e7d6640 1 Processor -- start 2024-03-29T10:56:30.211 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.230+0000 7f2d6e7d6640 1 -- start start 2024-03-29T10:56:30.211 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.230+0000 7f2d6e7d6640 10 monclient: init 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.231+0000 7f2d6e7d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.231+0000 7f2d6e7d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.231+0000 7f2d6e7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f2d680b0500 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.231+0000 7f2d6e7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f2d680b08e0 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.231+0000 7f2d6e7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f2d680aeca0 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.231+0000 7f2d677fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f2d680b08e0 0x7f2d6813d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46278/0 (socket says 172.21.15.5:46278) 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.231+0000 7f2d677fe640 1 -- 172.21.15.5:0/2961128719 learned_addr learned my addr 172.21.15.5:0/2961128719 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.231+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/2961128719 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d68140620 con 0x7f2d680b08e0 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/2961128719 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6813daf0 con 0x7f2d680aeca0 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/2961128719 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680ae180 con 0x7f2d680b0500 2024-03-29T10:56:30.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d6e7d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d6e7d6640 10 monclient(hunting): authenticate will time out at 1403.473404s 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203253165 0 0) 0x7f2d68140620 con 0x7f2d680b08e0 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 10 monclient(hunting): my global_id is 4503 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d50003520 con 0x7f2d680b08e0 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993425018 0 0) 0x7f2d680ae180 con 0x7f2d680b0500 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 10 monclient(hunting): my global_id is 4607 2024-03-29T10:56:30.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.232+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d50003840 con 0x7f2d680b0500 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.233+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428538255 0 0) 0x7f2d6813daf0 con 0x7f2d680aeca0 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.233+0000 7f2d66ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.233+0000 7f2d66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.233+0000 7f2d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.233+0000 7f2d66ffd640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.233+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d680ae180 con 0x7f2d680aeca0 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.233+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826477297 0 0) 0x7f2d680ae180 con 0x7f2d680aeca0 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 >> v1:172.21.15.5:6790/0 conn(0x7f2d680b0500 legacy=0x7f2d6813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 >> v1:172.21.15.5:6789/0 conn(0x7f2d680b08e0 legacy=0x7f2d6813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 1 monclient: found mon.noname-c 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:56:30.217 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680accf0 con 0x7f2d680aeca0 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2d600019b0 con 0x7f2d680aeca0 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: dump: 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:30.218 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d6e7d6640 5 monclient: authenticate success, global_id 4468 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d6e7d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d60001bd0 con 0x7f2d680aeca0 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 1 -- 172.21.15.5:0/2961128719 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2d60001f60 con 0x7f2d680aeca0 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d66ffd640 10 monclient: dump: 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:30.219 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:30.220 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:30.220 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:30.220 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:30.220 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:30.220 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:30.220 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:30.220 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:30.220 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:30.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d6e7d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:30.220 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.234+0000 7f2d6e7d6640 4 set_mon_vals no callback set 2024-03-29T10:56:30.228 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.235+0000 7f2d6e7d6640 10 monclient: shutdown 2024-03-29T10:56:30.228 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.235+0000 7f2d6e7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:30.228 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.235+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/2961128719 >> v1:172.21.15.177:6789/0 conn(0x7f2d680aeca0 legacy=0x7f2d6814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.228 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.235+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/2961128719 shutdown_connections 2024-03-29T10:56:30.228 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.235+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/2961128719 >> 172.21.15.5:0/2961128719 conn(0x7f2d6806b660 msgr2=0x7f2d6806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.235+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/2961128719 shutdown_connections 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.235+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/2961128719 wait complete. 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.236+0000 7f2d6e7d6640 10 monclient: build_initial_monmap 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.236+0000 7f2d6e7d6640 10 monclient: monmap: 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.237+0000 7f2d6e7d6640 1 librados: starting msgr at 2024-03-29T10:56:30.229 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.237+0000 7f2d6e7d6640 1 librados: starting objecter 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.237+0000 7f2d6e7d6640 1 Processor -- start 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.237+0000 7f2d6e7d6640 1 -- start start 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.237+0000 7f2d6e7d6640 1 librados: setting wanted keys 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.237+0000 7f2d6e7d6640 1 librados: calling monclient init 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.237+0000 7f2d6e7d6640 10 monclient: init 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6e7d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6e7d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6e7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f2d680aeca0 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6e7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f2d680b0500 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6e7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f2d6813a860 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6e7d6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d681dee70 con 0x7f2d680aeca0 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6e7d6640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d681270c0 con 0x7f2d680b0500 2024-03-29T10:56:30.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6e7d6640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d68127330 con 0x7f2d6813a860 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6e7d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6cd73640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f2d6813a860 0x7f2d681e6e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:38330/0 (socket says 172.21.15.5:38330) 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6e7d6640 10 monclient(hunting): authenticate will time out at 1403.479659s 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d6cd73640 1 -- 172.21.15.5:0/3776435301 learned_addr learned my addr 172.21.15.5:0/3776435301 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d64ff9640 10 client.?.objecter ms_handle_connect 0x7f2d6813a860 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d64ff9640 10 client.?.objecter ms_handle_connect 0x7f2d680aeca0 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.238+0000 7f2d64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310771604 0 0) 0x7f2d68127330 con 0x7f2d6813a860 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:30.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d54004eb0 con 0x7f2d6813a860 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 client.?.objecter ms_handle_connect 0x7f2d680b0500 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379209754 0 0) 0x7f2d681dee70 con 0x7f2d680aeca0 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d68127330 con 0x7f2d680aeca0 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891473651 0 0) 0x7f2d681270c0 con 0x7f2d680b0500 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:30.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d54005360 con 0x7f2d680b0500 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1262821742 0 0) 0x7f2d54004eb0 con 0x7f2d6813a860 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.239+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d681270c0 con 0x7f2d6813a860 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2d60001930 con 0x7f2d6813a860 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:30.233 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228642574 0 0) 0x7f2d68127330 con 0x7f2d680aeca0 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d54004eb0 con 0x7f2d680aeca0 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2d580018f0 con 0x7f2d680aeca0 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:30.234 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2376084914 0 0) 0x7f2d681270c0 con 0x7f2d6813a860 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 >> v1:172.21.15.177:6789/0 conn(0x7f2d680b0500 legacy=0x7f2d68139b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 >> v1:172.21.15.5:6789/0 conn(0x7f2d680aeca0 legacy=0x7f2d681df300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 1 monclient: found mon.c 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:30.235 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d681dea90 con 0x7f2d6813a860 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d6e7d6640 5 monclient: authenticate success, global_id 4610 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d6e7d6640 10 monclient: _renew_subs 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d6e7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/3776435301 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6806fd30 con 0x7f2d6813a860 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d6e7d6640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d6e7d6640 10 monclient: _renew_subs 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d6e7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/3776435301 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6813a290 con 0x7f2d6813a860 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.240+0000 7f2d6e7d6640 1 librados: init done 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.242+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d60003c20 con 0x7f2d6813a860 2024-03-29T10:56:30.236 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.242+0000 7f2d64ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.242+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f2d60003ff0 con 0x7f2d6813a860 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.242+0000 7f2d64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.242+0000 7f2d667fc640 4 set_mon_vals no callback set 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.242+0000 7f2d6e7d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.243+0000 7f2d64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.243+0000 7f2d64ff9640 10 monclient: dump: 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:30.237 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.244+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f2d60004210 con 0x7f2d6813a860 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.244+0000 7f2d6e7d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.244+0000 7f2d6e7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.244+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/3776435301 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d48005080 con 0x7f2d6813a860 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.245+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f2d60084ee0 con 0x7f2d6813a860 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.245+0000 7f2d64ff9640 10 client.4610.objecter ms_dispatch 0x7f2d68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.245+0000 7f2d64ff9640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.245+0000 7f2d64ff9640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.245+0000 7f2d64ff9640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.245+0000 7f2d67fff640 10 monclient: get_auth_request con 0x7f2d5406e300 auth_method 0 2024-03-29T10:56:30.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.245+0000 7f2d64ff9640 10 client.4610.objecter ms_handle_connect 0x7f2d5406e300 2024-03-29T10:56:30.239 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.252+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d600507c0 con 0x7f2d6813a860 2024-03-29T10:56:30.239 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.253+0000 7f2d64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:30.239 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.253+0000 7f2d64ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:30.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.621+0000 7f2d6e7d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:30.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.621+0000 7f2d6e7d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:30.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.621+0000 7f2d6e7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:30.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.621+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/3776435301 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d48005330 con 0x7f2d6813a860 2024-03-29T10:56:30.603 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.621+0000 7f2d64ff9640 1 -- 172.21.15.5:0/3776435301 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f2d600526f0 con 0x7f2d6813a860 2024-03-29T10:56:30.603 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.621+0000 7f2d64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:30.603 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.621+0000 7f2d64ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:30.603 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:56:30.603 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:30.605 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.625+0000 7f2d6e7d6640 10 librados: watch_flush enter 2024-03-29T10:56:30.605 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.625+0000 7f2d6e7d6640 10 librados: watch_flush exit 2024-03-29T10:56:30.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.625+0000 7f2d6e7d6640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T10:56:30.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.625+0000 7f2d6e7d6640 10 client.4610.objecter successfully canceled tick 2024-03-29T10:56:30.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.625+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/3776435301 >> v1:172.21.15.5:6816/2847500151 conn(0x7f2d5406e300 legacy=0x7f2d540707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.625+0000 7f2d6e7d6640 10 monclient: shutdown 2024-03-29T10:56:30.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.625+0000 7f2d6e7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:30.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.625+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/3776435301 >> v1:172.21.15.5:6790/0 conn(0x7f2d6813a860 legacy=0x7f2d681e6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.625+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/3776435301 shutdown_connections 2024-03-29T10:56:30.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.626+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/3776435301 >> 172.21.15.5:0/3776435301 conn(0x7f2d6806b660 msgr2=0x7f2d680a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:30.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.626+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/3776435301 shutdown_connections 2024-03-29T10:56:30.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.626+0000 7f2d6e7d6640 1 -- 172.21.15.5:0/3776435301 wait complete. 2024-03-29T10:56:30.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:30.626+0000 7f2d6e7d6640 1 librados: shutdown 2024-03-29T10:56:30.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:37.429 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:37.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.640+0000 7fcd5dc55640 10 monclient: get_monmap_and_config 2024-03-29T10:56:37.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.640+0000 7fcd5dc55640 10 monclient: build_initial_monmap 2024-03-29T10:56:37.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.640+0000 7fcd5dc55640 10 monclient: monmap: 2024-03-29T10:56:37.622 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:37.622 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:37.622 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:56:37.641358+0000 2024-03-29T10:56:37.622 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:56:37.641358+0000 2024-03-29T10:56:37.622 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:37.623 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:37.623 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:37.623 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:56:37.623 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:56:37.623 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:37.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.641+0000 7fcd5dc55640 1 Processor -- start 2024-03-29T10:56:37.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.641+0000 7fcd5dc55640 1 -- start start 2024-03-29T10:56:37.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.641+0000 7fcd5dc55640 10 monclient: init 2024-03-29T10:56:37.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.642+0000 7fcd5dc55640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:37.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.642+0000 7fcd5dc55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:37.624 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.642+0000 7fcd5dc55640 10 monclient(hunting): picked mon.noname-b con 0x7fcd580b0770 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:37.624 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.642+0000 7fcd5dc55640 10 monclient(hunting): picked mon.noname-c con 0x7fcd5814a700 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:37.624 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.642+0000 7fcd5dc55640 10 monclient(hunting): picked mon.noname-a con 0x7fcd580ac5a0 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:37.624 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.642+0000 7fcd5dc55640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd580aba70 con 0x7fcd580ac5a0 2024-03-29T10:56:37.624 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.642+0000 7fcd5dc55640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd580aecb0 con 0x7fcd5814a700 2024-03-29T10:56:37.624 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.642+0000 7fcd5dc55640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd580af150 con 0x7fcd580b0770 2024-03-29T10:56:37.624 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.642+0000 7fcd5dc55640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:37.624 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.642+0000 7fcd5dc55640 10 monclient(hunting): authenticate will time out at 1410.883751s 2024-03-29T10:56:37.626 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.643+0000 7fcd577fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fcd580b0770 0x7fcd58149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:34554/0 (socket says 172.21.15.5:34554) 2024-03-29T10:56:37.626 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.643+0000 7fcd577fe640 1 -- 172.21.15.5:0/2857014042 learned_addr learned my addr 172.21.15.5:0/2857014042 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:37.626 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.645+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076294889 0 0) 0x7fcd580af150 con 0x7fcd580b0770 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 10 monclient(hunting): my global_id is 4613 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd40003520 con 0x7fcd580b0770 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352592692 0 0) 0x7fcd580aba70 con 0x7fcd580ac5a0 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd580af150 con 0x7fcd580ac5a0 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3158359546 0 0) 0x7fcd40003520 con 0x7fcd580b0770 2024-03-29T10:56:37.627 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.646+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 >> v1:172.21.15.177:6789/0 conn(0x7fcd5814a700 legacy=0x7fcd580abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:37.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 >> v1:172.21.15.5:6789/0 conn(0x7fcd580ac5a0 legacy=0x7fcd58151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:37.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:37.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 1 monclient: found mon.noname-b 2024-03-29T10:56:37.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580ac9d0 con 0x7fcd580b0770 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fcd48001690 con 0x7fcd580b0770 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd567fc640 10 monclient: dump: 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:37.632 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:37.633 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:37.633 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:37.633 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:37.633 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:37.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd5dc55640 5 monclient: authenticate success, global_id 4613 2024-03-29T10:56:37.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.649+0000 7fcd5dc55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.651+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd480017f0 con 0x7fcd580b0770 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.651+0000 7fcd567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.651+0000 7fcd567fc640 1 -- 172.21.15.5:0/2857014042 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fcd48001c40 con 0x7fcd580b0770 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.651+0000 7fcd567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.651+0000 7fcd567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.651+0000 7fcd567fc640 10 monclient: dump: 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:37.635 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.651+0000 7fcd5dc55640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.651+0000 7fcd5dc55640 4 set_mon_vals no callback set 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.652+0000 7fcd5dc55640 10 monclient: shutdown 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.652+0000 7fcd5dc55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.652+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2857014042 >> v1:172.21.15.5:6790/0 conn(0x7fcd580b0770 legacy=0x7fcd58149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.652+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2857014042 shutdown_connections 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.652+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2857014042 >> 172.21.15.5:0/2857014042 conn(0x7fcd5806b660 msgr2=0x7fcd5806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.653+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2857014042 shutdown_connections 2024-03-29T10:56:37.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.656+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2857014042 wait complete. 2024-03-29T10:56:37.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.657+0000 7fcd5dc55640 10 monclient: build_initial_monmap 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.657+0000 7fcd5dc55640 10 monclient: monmap: 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.657+0000 7fcd5dc55640 1 librados: starting msgr at 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.657+0000 7fcd5dc55640 1 librados: starting objecter 2024-03-29T10:56:37.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.657+0000 7fcd5dc55640 1 Processor -- start 2024-03-29T10:56:37.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.658+0000 7fcd5dc55640 1 -- start start 2024-03-29T10:56:37.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.658+0000 7fcd5dc55640 1 librados: setting wanted keys 2024-03-29T10:56:37.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.658+0000 7fcd5dc55640 1 librados: calling monclient init 2024-03-29T10:56:37.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.658+0000 7fcd5dc55640 10 monclient: init 2024-03-29T10:56:37.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.659+0000 7fcd5dc55640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:37.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.659+0000 7fcd5dc55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:37.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.659+0000 7fcd5dc55640 10 monclient(hunting): picked mon.noname-a con 0x7fcd580ac5a0 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:37.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.659+0000 7fcd5dc55640 10 monclient(hunting): picked mon.noname-c con 0x7fcd580b0770 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:37.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.659+0000 7fcd5dc55640 10 monclient(hunting): picked mon.noname-b con 0x7fcd5814a700 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:37.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.659+0000 7fcd5dc55640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd58128460 con 0x7fcd580ac5a0 2024-03-29T10:56:37.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.659+0000 7fcd5dc55640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd581263b0 con 0x7fcd5814a700 2024-03-29T10:56:37.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.659+0000 7fcd5dc55640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd58145960 con 0x7fcd580b0770 2024-03-29T10:56:37.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.659+0000 7fcd5dc55640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:37.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.659+0000 7fcd5dc55640 10 monclient(hunting): authenticate will time out at 1410.900372s 2024-03-29T10:56:37.642 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.661+0000 7fcd56ffd640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fcd580b0770 0x7fcd58144b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:39072/0 (socket says 172.21.15.5:39072) 2024-03-29T10:56:37.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.661+0000 7fcd56ffd640 1 -- 172.21.15.5:0/2311643390 learned_addr learned my addr 172.21.15.5:0/2311643390 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:37.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.661+0000 7fcd3ffff640 10 client.?.objecter ms_handle_connect 0x7fcd580b0770 2024-03-29T10:56:37.643 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.661+0000 7fcd3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.662+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201457200 0 0) 0x7fcd58145960 con 0x7fcd580b0770 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.662+0000 7fcd3ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.662+0000 7fcd3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.662+0000 7fcd3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 10 monclient(hunting): my global_id is 4616 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd44004eb0 con 0x7fcd580b0770 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 10 client.?.objecter ms_handle_connect 0x7fcd580ac5a0 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56482504 0 0) 0x7fcd58128460 con 0x7fcd580ac5a0 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:37.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 10 monclient(hunting): my global_id is 4512 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd58145960 con 0x7fcd580ac5a0 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.663+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2384503848 0 0) 0x7fcd58145960 con 0x7fcd580ac5a0 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd58128460 con 0x7fcd580ac5a0 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fcd48001b00 con 0x7fcd580ac5a0 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 10 monclient(hunting): dump: 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:37.650 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 352301905 0 0) 0x7fcd44004eb0 con 0x7fcd580b0770 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd58145960 con 0x7fcd580b0770 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 719599364 0 0) 0x7fcd58128460 con 0x7fcd580ac5a0 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 >> v1:172.21.15.5:6790/0 conn(0x7fcd580b0770 legacy=0x7fcd58144b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 >> v1:172.21.15.177:6789/0 conn(0x7fcd5814a700 legacy=0x7fcd58145250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 1 monclient: found mon.a 2024-03-29T10:56:37.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd581df8b0 con 0x7fcd580ac5a0 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 10 monclient: _finish_auth 0 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd5dc55640 5 monclient: authenticate success, global_id 4512 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd5dc55640 10 monclient: _renew_subs 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd5dc55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.664+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2311643390 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd581dfb00 con 0x7fcd580ac5a0 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd5dc55640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd5dc55640 10 monclient: _renew_subs 2024-03-29T10:56:37.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd5dc55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:56:37.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2311643390 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd58125fe0 con 0x7fcd580ac5a0 2024-03-29T10:56:37.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd5dc55640 1 librados: init done 2024-03-29T10:56:37.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd48001c60 con 0x7fcd580ac5a0 2024-03-29T10:56:37.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:37.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fcd48004d90 con 0x7fcd580ac5a0 2024-03-29T10:56:37.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:37.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:37.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd3ffff640 10 monclient: dump: 2024-03-29T10:56:37.654 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:37.654 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:37.654 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:37.655 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:37.655 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:37.655 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:37.655 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:37.655 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:37.655 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:37.656 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:37.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.665+0000 7fcd557fa640 4 set_mon_vals no callback set 2024-03-29T10:56:37.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.667+0000 7fcd5dc55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:37.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.667+0000 7fcd5dc55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:37.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.667+0000 7fcd5dc55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:56:37.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.667+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2311643390 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd38005080 con 0x7fcd580ac5a0 2024-03-29T10:56:37.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.667+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fcd4801ad60 con 0x7fcd580ac5a0 2024-03-29T10:56:37.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.668+0000 7fcd56ffd640 10 monclient: get_auth_request con 0x7fcd4406dd80 auth_method 0 2024-03-29T10:56:37.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.668+0000 7fcd3ffff640 10 client.4512.objecter ms_handle_connect 0x7fcd4406dd80 2024-03-29T10:56:37.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.669+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fcd48054810 con 0x7fcd580ac5a0 2024-03-29T10:56:37.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.669+0000 7fcd3ffff640 10 client.4512.objecter ms_dispatch 0x7fcd58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:37.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.669+0000 7fcd3ffff640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:37.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.669+0000 7fcd3ffff640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:37.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.669+0000 7fcd3ffff640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T10:56:37.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.677+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcd48050320 con 0x7fcd580ac5a0 2024-03-29T10:56:37.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.677+0000 7fcd3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:37.658 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:37.677+0000 7fcd3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:38.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.028+0000 7fcd5dc55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:38.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.028+0000 7fcd5dc55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:38.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.028+0000 7fcd5dc55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:56:38.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.028+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2311643390 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd38005330 con 0x7fcd580ac5a0 2024-03-29T10:56:38.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.029+0000 7fcd3ffff640 1 -- 172.21.15.5:0/2311643390 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fcd48084090 con 0x7fcd580ac5a0 2024-03-29T10:56:38.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.029+0000 7fcd3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:38.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.029+0000 7fcd3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:38.011 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:56:38.012 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:38.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 10 librados: watch_flush enter 2024-03-29T10:56:38.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 10 librados: watch_flush exit 2024-03-29T10:56:38.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T10:56:38.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 10 client.4512.objecter successfully canceled tick 2024-03-29T10:56:38.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2311643390 >> v1:172.21.15.5:6816/2847500151 conn(0x7fcd4406dd80 legacy=0x7fcd44070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:38.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 10 monclient: shutdown 2024-03-29T10:56:38.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:38.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2311643390 >> v1:172.21.15.5:6789/0 conn(0x7fcd580ac5a0 legacy=0x7fcd581268d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:38.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2311643390 shutdown_connections 2024-03-29T10:56:38.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2311643390 >> 172.21.15.5:0/2311643390 conn(0x7fcd5806b660 msgr2=0x7fcd580713f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:38.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2311643390 shutdown_connections 2024-03-29T10:56:38.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.033+0000 7fcd5dc55640 1 -- 172.21.15.5:0/2311643390 wait complete. 2024-03-29T10:56:38.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:38.034+0000 7fcd5dc55640 1 librados: shutdown 2024-03-29T10:56:38.028 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:44.835 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:44.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:44.998+0000 7fbffcff3640 10 monclient: get_monmap_and_config 2024-03-29T10:56:44.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:44.998+0000 7fbffcff3640 10 monclient: build_initial_monmap 2024-03-29T10:56:44.979 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:44.998+0000 7fbffcff3640 10 monclient: monmap: 2024-03-29T10:56:44.979 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:44.980 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:44.980 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:56:44.999655+0000 2024-03-29T10:56:44.980 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:56:44.999655+0000 2024-03-29T10:56:44.980 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:44.980 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:44.980 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:44.980 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:56:44.980 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:56:44.980 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:44.982 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.002+0000 7fbffcff3640 1 Processor -- start 2024-03-29T10:56:44.983 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.002+0000 7fbffcff3640 1 -- start start 2024-03-29T10:56:44.983 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.002+0000 7fbffcff3640 10 monclient: init 2024-03-29T10:56:44.986 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbffcff3640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:44.986 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbffcff3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:44.986 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbffcff3640 10 monclient(hunting): picked mon.noname-c con 0x7fbff80b2790 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:44.986 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbffcff3640 10 monclient(hunting): picked mon.noname-b con 0x7fbff8143c80 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:44.986 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbffcff3640 10 monclient(hunting): picked mon.noname-a con 0x7fbff8146e30 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:44.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbffcff3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff8146290 con 0x7fbff8146e30 2024-03-29T10:56:44.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbffcff3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff80b3970 con 0x7fbff80b2790 2024-03-29T10:56:44.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbffcff3640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff806ee80 con 0x7fbff8143c80 2024-03-29T10:56:44.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbffcff3640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:44.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbffcff3640 10 monclient(hunting): authenticate will time out at 1418.244401s 2024-03-29T10:56:44.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbff6ffd640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fbff8143c80 0x7fbff8146720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:53354/0 (socket says 172.21.15.5:53354) 2024-03-29T10:56:44.987 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.003+0000 7fbff6ffd640 1 -- 172.21.15.5:0/153583828 learned_addr learned my addr 172.21.15.5:0/153583828 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:44.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.006+0000 7fbff67fc640 1 -- 172.21.15.5:0/153583828 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222971237 0 0) 0x7fbff806ee80 con 0x7fbff8143c80 2024-03-29T10:56:44.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.006+0000 7fbff67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:44.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.006+0000 7fbff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:44.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.006+0000 7fbff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:44.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.006+0000 7fbff67fc640 10 monclient(hunting): my global_id is 4619 2024-03-29T10:56:44.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.006+0000 7fbff67fc640 1 -- 172.21.15.5:0/153583828 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe4003520 con 0x7fbff8143c80 2024-03-29T10:56:44.988 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.007+0000 7fbff67fc640 1 -- 172.21.15.5:0/153583828 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439457663 0 0) 0x7fbfe4003520 con 0x7fbff8143c80 2024-03-29T10:56:44.989 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.007+0000 7fbff67fc640 1 -- 172.21.15.5:0/153583828 >> v1:172.21.15.177:6789/0 conn(0x7fbff80b2790 legacy=0x7fbff8143570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:44.994 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 1 -- 172.21.15.5:0/153583828 >> v1:172.21.15.5:6789/0 conn(0x7fbff8146e30 legacy=0x7fbff8151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 1 monclient: found mon.noname-b 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 1 -- 172.21.15.5:0/153583828 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff806f160 con 0x7fbff8143c80 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 1 -- 172.21.15.5:0/153583828 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbfe0001a10 con 0x7fbff8143c80 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:44.995 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbff67fc640 10 monclient: dump: 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbffcff3640 5 monclient: authenticate success, global_id 4619 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.009+0000 7fbffcff3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.010+0000 7fbff67fc640 1 -- 172.21.15.5:0/153583828 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfe0001be0 con 0x7fbff8143c80 2024-03-29T10:56:44.996 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.010+0000 7fbff67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.010+0000 7fbff67fc640 1 -- 172.21.15.5:0/153583828 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbfe0002030 con 0x7fbff8143c80 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.010+0000 7fbff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.010+0000 7fbff67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.010+0000 7fbff67fc640 10 monclient: dump: 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:44.997 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.011+0000 7fbffcff3640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.011+0000 7fbffcff3640 4 set_mon_vals no callback set 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.012+0000 7fbffcff3640 10 monclient: shutdown 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.012+0000 7fbffcff3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.012+0000 7fbffcff3640 1 -- 172.21.15.5:0/153583828 >> v1:172.21.15.5:6790/0 conn(0x7fbff8143c80 legacy=0x7fbff8146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.012+0000 7fbffcff3640 1 -- 172.21.15.5:0/153583828 shutdown_connections 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.012+0000 7fbffcff3640 1 -- 172.21.15.5:0/153583828 >> 172.21.15.5:0/153583828 conn(0x7fbff806d070 msgr2=0x7fbff8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.012+0000 7fbffcff3640 1 -- 172.21.15.5:0/153583828 shutdown_connections 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.013+0000 7fbffcff3640 1 -- 172.21.15.5:0/153583828 wait complete. 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.013+0000 7fbffcff3640 10 monclient: build_initial_monmap 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.013+0000 7fbffcff3640 10 monclient: monmap: 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:44.998 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:56:44.999 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:56:44.999 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:44.999 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:44.999 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:44.999 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:56:44.999 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:56:44.999 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:44.999 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.014+0000 7fbffcff3640 1 librados: starting msgr at 2024-03-29T10:56:44.999 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.014+0000 7fbffcff3640 1 librados: starting objecter 2024-03-29T10:56:44.999 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.014+0000 7fbffcff3640 1 Processor -- start 2024-03-29T10:56:45.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.014+0000 7fbffcff3640 1 -- start start 2024-03-29T10:56:45.005 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.014+0000 7fbffcff3640 1 librados: setting wanted keys 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.014+0000 7fbffcff3640 1 librados: calling monclient init 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.014+0000 7fbffcff3640 10 monclient: init 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbffcff3640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbffcff3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbffcff3640 10 monclient(hunting): picked mon.noname-a con 0x7fbff80b2790 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbffcff3640 10 monclient(hunting): picked mon.noname-c con 0x7fbff8146e30 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbffcff3640 10 monclient(hunting): picked mon.noname-b con 0x7fbff80b6570 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbffcff3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff8149770 con 0x7fbff80b2790 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbffcff3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff80b6b20 con 0x7fbff80b6570 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbffcff3640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff8148250 con 0x7fbff8146e30 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbffcff3640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:45.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbffcff3640 10 monclient(hunting): authenticate will time out at 1418.256506s 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbff6ffd640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fbff8146e30 0x7fbff8148950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:57340/0 (socket says 172.21.15.5:57340) 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbff6ffd640 1 -- 172.21.15.5:0/84693891 learned_addr learned my addr 172.21.15.5:0/84693891 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbfeffff640 10 client.?.objecter ms_handle_connect 0x7fbff8146e30 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbfeffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209481101 0 0) 0x7fbff8148250 con 0x7fbff8146e30 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbfeffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbfeffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbfeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbfeffff640 10 monclient(hunting): my global_id is 4622 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.015+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd8004eb0 con 0x7fbff8146e30 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.016+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908316439 0 0) 0x7fbfd8004eb0 con 0x7fbff8146e30 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.016+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbff8148250 con 0x7fbff8146e30 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.016+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbfe00013b0 con 0x7fbff8146e30 2024-03-29T10:56:45.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.016+0000 7fbfeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.016+0000 7fbfeffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.016+0000 7fbfeffff640 10 monclient(hunting): dump: 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.016+0000 7fbfeffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.016+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2025573360 0 0) 0x7fbff8148250 con 0x7fbff8146e30 2024-03-29T10:56:45.008 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.016+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 >> v1:172.21.15.177:6789/0 conn(0x7fbff80b6570 legacy=0x7fbff8149060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.019+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 >> v1:172.21.15.5:6789/0 conn(0x7fbff80b2790 legacy=0x7fbff80b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.019+0000 7fbfeffff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.019+0000 7fbfeffff640 1 monclient: found mon.c 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.019+0000 7fbfeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.019+0000 7fbfeffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.019+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff81473c0 con 0x7fbff8146e30 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.019+0000 7fbfeffff640 10 monclient: _finish_auth 0 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.019+0000 7fbfeffff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.020+0000 7fbfeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.020+0000 7fbffcff3640 5 monclient: authenticate success, global_id 4622 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.020+0000 7fbffcff3640 10 monclient: _renew_subs 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.020+0000 7fbffcff3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.020+0000 7fbffcff3640 1 -- 172.21.15.5:0/84693891 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff81e70f0 con 0x7fbff8146e30 2024-03-29T10:56:45.009 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.020+0000 7fbffcff3640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.020+0000 7fbffcff3640 10 monclient: _renew_subs 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.020+0000 7fbffcff3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.020+0000 7fbffcff3640 1 -- 172.21.15.5:0/84693891 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff81e7460 con 0x7fbff8146e30 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.020+0000 7fbffcff3640 1 librados: init done 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.021+0000 7fbffcff3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.021+0000 7fbffcff3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.021+0000 7fbffcff3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.021+0000 7fbffcff3640 1 -- 172.21.15.5:0/84693891 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff80b1e70 con 0x7fbff8146e30 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.021+0000 7fbfeffff640 10 client.4622.objecter ms_handle_connect 0x7fbff80b2790 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.021+0000 7fbfeffff640 10 client.4622.objecter resend_mon_ops 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.021+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfe0001fb0 con 0x7fbff8146e30 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.021+0000 7fbfeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.021+0000 7fbff5ffb640 4 set_mon_vals no callback set 2024-03-29T10:56:45.010 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.022+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fbfe0005110 con 0x7fbff8146e30 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.022+0000 7fbfeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.022+0000 7fbfeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.022+0000 7fbfeffff640 10 monclient: dump: 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:45.011 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.023+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fbfe001b0e0 con 0x7fbff8146e30 2024-03-29T10:56:45.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.025+0000 7fbff77fe640 10 monclient: get_auth_request con 0x7fbfd806db30 auth_method 0 2024-03-29T10:56:45.012 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.026+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fbfe0084f50 con 0x7fbff8146e30 2024-03-29T10:56:45.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.026+0000 7fbfeffff640 10 client.4622.objecter ms_dispatch 0x7fbff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:45.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.026+0000 7fbfeffff640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:45.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.026+0000 7fbfeffff640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:45.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.026+0000 7fbfeffff640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T10:56:45.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.026+0000 7fbfeffff640 10 client.4622.objecter ms_handle_connect 0x7fbfd806db30 2024-03-29T10:56:45.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.034+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbfe0050830 con 0x7fbff8146e30 2024-03-29T10:56:45.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.034+0000 7fbfeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:45.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.034+0000 7fbfeffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:45.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.360+0000 7fbffcff3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:45.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.360+0000 7fbffcff3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:45.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.360+0000 7fbffcff3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:45.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.360+0000 7fbffcff3640 1 -- 172.21.15.5:0/84693891 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbff804f730 con 0x7fbff8146e30 2024-03-29T10:56:45.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.361+0000 7fbfeffff640 1 -- 172.21.15.5:0/84693891 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fbfe0052760 con 0x7fbff8146e30 2024-03-29T10:56:45.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.361+0000 7fbfeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:45.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.361+0000 7fbfeffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:45.342 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:56:45.342 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:45.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.364+0000 7fbffcff3640 10 librados: watch_flush enter 2024-03-29T10:56:45.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.364+0000 7fbffcff3640 10 librados: watch_flush exit 2024-03-29T10:56:45.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.364+0000 7fbffcff3640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T10:56:45.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.364+0000 7fbffcff3640 10 client.4622.objecter successfully canceled tick 2024-03-29T10:56:45.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.364+0000 7fbffcff3640 1 -- 172.21.15.5:0/84693891 >> v1:172.21.15.5:6816/2847500151 conn(0x7fbfd806db30 legacy=0x7fbfd806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:45.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.364+0000 7fbffcff3640 10 monclient: shutdown 2024-03-29T10:56:45.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.364+0000 7fbffcff3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:45.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.364+0000 7fbffcff3640 1 -- 172.21.15.5:0/84693891 >> v1:172.21.15.5:6790/0 conn(0x7fbff8146e30 legacy=0x7fbff8148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:45.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.365+0000 7fbffcff3640 1 -- 172.21.15.5:0/84693891 shutdown_connections 2024-03-29T10:56:45.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.365+0000 7fbffcff3640 1 -- 172.21.15.5:0/84693891 >> 172.21.15.5:0/84693891 conn(0x7fbff806d070 msgr2=0x7fbff8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:45.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.365+0000 7fbffcff3640 1 -- 172.21.15.5:0/84693891 shutdown_connections 2024-03-29T10:56:45.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.365+0000 7fbffcff3640 1 -- 172.21.15.5:0/84693891 wait complete. 2024-03-29T10:56:45.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:45.365+0000 7fbffcff3640 1 librados: shutdown 2024-03-29T10:56:45.363 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:52.170 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:52.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.333+0000 7f33a359e640 10 monclient: get_monmap_and_config 2024-03-29T10:56:52.315 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.333+0000 7f33a359e640 10 monclient: build_initial_monmap 2024-03-29T10:56:52.315 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.333+0000 7f33a359e640 10 monclient: monmap: 2024-03-29T10:56:52.315 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:52.315 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:52.315 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:56:52.334872+0000 2024-03-29T10:56:52.315 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:56:52.334872+0000 2024-03-29T10:56:52.315 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:52.316 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:52.316 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:52.316 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:56:52.316 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:56:52.316 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:52.318 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.334+0000 7f33a359e640 1 Processor -- start 2024-03-29T10:56:52.318 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.334+0000 7f33a359e640 1 -- start start 2024-03-29T10:56:52.318 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.334+0000 7f33a359e640 10 monclient: init 2024-03-29T10:56:52.318 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.335+0000 7f33a359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:52.319 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.335+0000 7f33a359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:52.319 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.335+0000 7f33a359e640 10 monclient(hunting): picked mon.noname-c con 0x7f33a406bf30 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:52.319 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.335+0000 7f33a359e640 10 monclient(hunting): picked mon.noname-a con 0x7f33a40b0b30 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:52.319 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.335+0000 7f33a359e640 10 monclient(hunting): picked mon.noname-b con 0x7f33a40aea80 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:52.319 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.336+0000 7f33a359e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a406c310 con 0x7f33a40b0b30 2024-03-29T10:56:52.319 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.336+0000 7f33a359e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a40ad990 con 0x7f33a406bf30 2024-03-29T10:56:52.319 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.336+0000 7f33a359e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a40adc70 con 0x7f33a40aea80 2024-03-29T10:56:52.319 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.336+0000 7f33a359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:52.320 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.336+0000 7f33a359e640 10 monclient(hunting): authenticate will time out at 1425.577113s 2024-03-29T10:56:52.320 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.337+0000 7f33a259c640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f33a406bf30 0x7f33a40b0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:34128/0 (socket says 172.21.15.5:34128) 2024-03-29T10:56:52.320 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.337+0000 7f33a259c640 1 -- 172.21.15.5:0/2361739032 learned_addr learned my addr 172.21.15.5:0/2361739032 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:52.320 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.338+0000 7f33a159a640 1 -- 172.21.15.5:0/2361739032 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146147728 0 0) 0x7f33a40ad990 con 0x7f33a406bf30 2024-03-29T10:56:52.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.338+0000 7f33a159a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:52.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.338+0000 7f33a159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:52.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.338+0000 7f33a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:52.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.338+0000 7f33a159a640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:56:52.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.338+0000 7f33a159a640 1 -- 172.21.15.5:0/2361739032 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f338c003520 con 0x7f33a406bf30 2024-03-29T10:56:52.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 1 -- 172.21.15.5:0/2361739032 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127648932 0 0) 0x7f33a40adc70 con 0x7f33a40aea80 2024-03-29T10:56:52.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:52.327 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 1 -- 172.21.15.5:0/2361739032 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a40ad990 con 0x7f33a40aea80 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 1 -- 172.21.15.5:0/2361739032 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4249387280 0 0) 0x7f338c003520 con 0x7f33a406bf30 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 1 -- 172.21.15.5:0/2361739032 >> v1:172.21.15.5:6790/0 conn(0x7f33a40aea80 legacy=0x7f33a414dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 1 -- 172.21.15.5:0/2361739032 >> v1:172.21.15.5:6789/0 conn(0x7f33a40b0b30 legacy=0x7f33a40ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 10 monclient: _finish_hunting 0 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 1 monclient: found mon.noname-c 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.341+0000 7f33a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 1 -- 172.21.15.5:0/2361739032 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a40adf50 con 0x7f33a406bf30 2024-03-29T10:56:52.328 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 10 monclient: _finish_auth 0 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 10 monclient: _check_auth_tickets 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a359e640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 1 -- 172.21.15.5:0/2361739032 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f3394001610 con 0x7f33a406bf30 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 10 monclient: dump: 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:52.329 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:52.330 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:52.330 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:52.330 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:52.330 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:52.330 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:52.330 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:52.330 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 1 -- 172.21.15.5:0/2361739032 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3394001830 con 0x7f33a406bf30 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a359e640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.342+0000 7f33a359e640 4 set_mon_vals no callback set 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.343+0000 7f33a359e640 10 monclient: shutdown 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.343+0000 7f33a359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.343+0000 7f33a359e640 1 -- 172.21.15.5:0/2361739032 >> v1:172.21.15.177:6789/0 conn(0x7f33a406bf30 legacy=0x7f33a40b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.344+0000 7f33a359e640 1 -- 172.21.15.5:0/2361739032 shutdown_connections 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.344+0000 7f33a359e640 1 -- 172.21.15.5:0/2361739032 >> 172.21.15.5:0/2361739032 conn(0x7f33a40672a0 msgr2=0x7f33a40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.345+0000 7f33a359e640 1 -- 172.21.15.5:0/2361739032 shutdown_connections 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.345+0000 7f33a359e640 1 -- 172.21.15.5:0/2361739032 wait complete. 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.346+0000 7f33a359e640 10 monclient: build_initial_monmap 2024-03-29T10:56:52.331 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.346+0000 7f33a359e640 10 monclient: monmap: 2024-03-29T10:56:52.332 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:52.332 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:52.332 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:56:52.332 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:56:52.332 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:52.332 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:52.332 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:52.332 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:56:52.332 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:56:52.333 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:52.333 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.346+0000 7f33a359e640 1 librados: starting msgr at 2024-03-29T10:56:52.333 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.346+0000 7f33a359e640 1 librados: starting objecter 2024-03-29T10:56:52.333 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 1 Processor -- start 2024-03-29T10:56:52.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 1 -- start start 2024-03-29T10:56:52.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 1 librados: setting wanted keys 2024-03-29T10:56:52.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 1 librados: calling monclient init 2024-03-29T10:56:52.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 10 monclient: init 2024-03-29T10:56:52.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:52.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:52.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 10 monclient(hunting): picked mon.noname-a con 0x7f33a406bf30 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:52.340 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 10 monclient(hunting): picked mon.noname-c con 0x7f33a40aea80 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 10 monclient(hunting): picked mon.noname-b con 0x7f33a40b0b30 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a41aaf30 con 0x7f33a406bf30 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a40ab600 con 0x7f33a40b0b30 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a40ab790 con 0x7f33a40aea80 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.347+0000 7f33a359e640 10 monclient(hunting): authenticate will time out at 1425.589049s 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.348+0000 7f33a2d9d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f33a40b0b30 0x7f33a41db890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:34562/0 (socket says 172.21.15.5:34562) 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.348+0000 7f33a2d9d640 1 -- 172.21.15.5:0/2036093515 learned_addr learned my addr 172.21.15.5:0/2036093515 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.349+0000 7f3392ffd640 10 client.?.objecter ms_handle_connect 0x7f33a40b0b30 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.349+0000 7f3392ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.349+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585275143 0 0) 0x7f33a40ab600 con 0x7f33a40b0b30 2024-03-29T10:56:52.341 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.349+0000 7f3392ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:52.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.349+0000 7f3392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:52.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.349+0000 7f3392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:52.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.349+0000 7f3392ffd640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:56:52.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.349+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3384004eb0 con 0x7f33a40b0b30 2024-03-29T10:56:52.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.350+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3871933783 0 0) 0x7f3384004eb0 con 0x7f33a40b0b30 2024-03-29T10:56:52.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.350+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33a40ab600 con 0x7f33a40b0b30 2024-03-29T10:56:52.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.350+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f339c001d90 con 0x7f33a40b0b30 2024-03-29T10:56:52.342 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.350+0000 7f3392ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:52.343 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.350+0000 7f3392ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:52.343 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.350+0000 7f3392ffd640 10 monclient(hunting): dump: 2024-03-29T10:56:52.343 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:52.343 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:52.343 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:52.343 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:52.343 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:52.343 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.350+0000 7f3392ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.351+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2757491265 0 0) 0x7f33a40ab600 con 0x7f33a40b0b30 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.351+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 >> v1:172.21.15.5:6790/0 conn(0x7f33a40aea80 legacy=0x7f33a41db180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.352+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 >> v1:172.21.15.5:6789/0 conn(0x7f33a406bf30 legacy=0x7f33a40ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.352+0000 7f3392ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.352+0000 7f3392ffd640 1 monclient: found mon.b 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.352+0000 7f3392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:52.344 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.352+0000 7f3392ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.352+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a40ab920 con 0x7f33a40b0b30 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.352+0000 7f3392ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.352+0000 7f3392ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f3392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f33a359e640 5 monclient: authenticate success, global_id 4477 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f33a359e640 10 monclient: _renew_subs 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f33a359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f33a359e640 1 -- 172.21.15.5:0/2036093515 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a41e6c40 con 0x7f33a40b0b30 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f33a359e640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f33a359e640 10 monclient: _renew_subs 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f33a359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f33a359e640 1 -- 172.21.15.5:0/2036093515 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a41e6fb0 con 0x7f33a40b0b30 2024-03-29T10:56:52.345 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f33a359e640 1 librados: init done 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f339c001ff0 con 0x7f33a40b0b30 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f3392ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f339c003e20 con 0x7f33a40b0b30 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f3392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f3392ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.353+0000 7f3392ffd640 10 monclient: dump: 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:52.346 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.354+0000 7f33a0d99640 4 set_mon_vals no callback set 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.354+0000 7f33a359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.355+0000 7f33a359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.356+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f339c004060 con 0x7f33a40b0b30 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.356+0000 7f33a359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.356+0000 7f33a359e640 1 -- 172.21.15.5:0/2036093515 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3388005080 con 0x7f33a40b0b30 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.356+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f339c084fe0 con 0x7f33a40b0b30 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.356+0000 7f3392ffd640 10 client.4477.objecter ms_dispatch 0x7f33a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:52.347 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.356+0000 7f3392ffd640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:52.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.356+0000 7f3392ffd640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:52.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.356+0000 7f3392ffd640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T10:56:52.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.357+0000 7f33a259c640 10 monclient: get_auth_request con 0x7f338406db90 auth_method 0 2024-03-29T10:56:52.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.357+0000 7f3392ffd640 10 client.4477.objecter ms_handle_connect 0x7f338406db90 2024-03-29T10:56:52.348 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.364+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f339c0508c0 con 0x7f33a40b0b30 2024-03-29T10:56:52.349 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.365+0000 7f3392ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:52.349 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.365+0000 7f3392ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:52.686 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.706+0000 7f33a359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:52.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.706+0000 7f33a359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:52.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.706+0000 7f33a359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:56:52.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.706+0000 7f33a359e640 1 -- 172.21.15.5:0/2036093515 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3388005330 con 0x7f33a40b0b30 2024-03-29T10:56:52.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.707+0000 7f3392ffd640 1 -- 172.21.15.5:0/2036093515 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f339c0527f0 con 0x7f33a40b0b30 2024-03-29T10:56:52.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.707+0000 7f3392ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:52.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.707+0000 7f3392ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:52.689 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:56:52.689 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:52.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 10 librados: watch_flush enter 2024-03-29T10:56:52.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 10 librados: watch_flush exit 2024-03-29T10:56:52.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T10:56:52.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 10 client.4477.objecter successfully canceled tick 2024-03-29T10:56:52.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 1 -- 172.21.15.5:0/2036093515 >> v1:172.21.15.5:6816/2847500151 conn(0x7f338406db90 legacy=0x7f3384070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:52.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 10 monclient: shutdown 2024-03-29T10:56:52.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:52.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 1 -- 172.21.15.5:0/2036093515 >> v1:172.21.15.177:6789/0 conn(0x7f33a40b0b30 legacy=0x7f33a41db890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:52.692 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 1 -- 172.21.15.5:0/2036093515 shutdown_connections 2024-03-29T10:56:52.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 1 -- 172.21.15.5:0/2036093515 >> 172.21.15.5:0/2036093515 conn(0x7f33a40672a0 msgr2=0x7f33a4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:52.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 1 -- 172.21.15.5:0/2036093515 shutdown_connections 2024-03-29T10:56:52.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 1 -- 172.21.15.5:0/2036093515 wait complete. 2024-03-29T10:56:52.693 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:52.711+0000 7f33a359e640 1 librados: shutdown 2024-03-29T10:56:52.709 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:59.516 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:59.665 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.684+0000 7fd43aab3640 10 monclient: get_monmap_and_config 2024-03-29T10:56:59.665 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.684+0000 7fd43aab3640 10 monclient: build_initial_monmap 2024-03-29T10:56:59.666 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.685+0000 7fd43aab3640 10 monclient: monmap: 2024-03-29T10:56:59.666 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:59.666 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:59.666 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:56:59.686004+0000 2024-03-29T10:56:59.666 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:56:59.686004+0000 2024-03-29T10:56:59.666 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:59.666 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:59.667 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:59.667 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:56:59.667 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:56:59.667 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:59.667 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.685+0000 7fd43aab3640 1 Processor -- start 2024-03-29T10:56:59.668 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 1 -- start start 2024-03-29T10:56:59.668 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 10 monclient: init 2024-03-29T10:56:59.668 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:59.668 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:59.668 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 10 monclient(hunting): picked mon.noname-b con 0x7fd4341457f0 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:59.669 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 10 monclient(hunting): picked mon.noname-a con 0x7fd43413f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:59.669 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 10 monclient(hunting): picked mon.noname-c con 0x7fd4340a9d70 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:59.669 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4340aaef0 con 0x7fd43413f650 2024-03-29T10:56:59.669 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd434145bd0 con 0x7fd4340a9d70 2024-03-29T10:56:59.669 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd434145eb0 con 0x7fd4341457f0 2024-03-29T10:56:59.669 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:59.669 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.686+0000 7fd43aab3640 10 monclient(hunting): authenticate will time out at 1432.928006s 2024-03-29T10:56:59.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.687+0000 7fd43884f640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fd4341457f0 0x7fd43413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:51056/0 (socket says 172.21.15.5:51056) 2024-03-29T10:56:59.670 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.687+0000 7fd43884f640 1 -- 172.21.15.5:0/1359234645 learned_addr learned my addr 172.21.15.5:0/1359234645 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:59.671 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.688+0000 7fd4337fe640 1 -- 172.21.15.5:0/1359234645 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379274879 0 0) 0x7fd434145eb0 con 0x7fd4341457f0 2024-03-29T10:56:59.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.688+0000 7fd4337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:59.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.688+0000 7fd4337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:59.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.688+0000 7fd4337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:59.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.688+0000 7fd4337fe640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:56:59.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.688+0000 7fd4337fe640 1 -- 172.21.15.5:0/1359234645 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd424003520 con 0x7fd4341457f0 2024-03-29T10:56:59.672 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.689+0000 7fd4337fe640 1 -- 172.21.15.5:0/1359234645 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2608109086 0 0) 0x7fd424003520 con 0x7fd4341457f0 2024-03-29T10:56:59.673 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.689+0000 7fd4337fe640 1 -- 172.21.15.5:0/1359234645 >> v1:172.21.15.177:6789/0 conn(0x7fd4340a9d70 legacy=0x7fd43414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:59.673 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 1 -- 172.21.15.5:0/1359234645 >> v1:172.21.15.5:6789/0 conn(0x7fd43413f650 legacy=0x7fd4340b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:59.680 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:59.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 1 monclient: found mon.noname-b 2024-03-29T10:56:59.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:59.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:56:59.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 1 -- 172.21.15.5:0/1359234645 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd434146340 con 0x7fd4341457f0 2024-03-29T10:56:59.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:59.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:59.681 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:59.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 1 -- 172.21.15.5:0/1359234645 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fd428001690 con 0x7fd4341457f0 2024-03-29T10:56:59.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:59.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:59.682 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.690+0000 7fd4337fe640 10 monclient: dump: 2024-03-29T10:56:59.682 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:59.682 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:59.682 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:59.682 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:59.682 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:59.683 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:59.683 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:59.683 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:59.683 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:59.683 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:59.683 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.691+0000 7fd43aab3640 5 monclient: authenticate success, global_id 4628 2024-03-29T10:56:59.683 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.691+0000 7fd43aab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:59.683 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.693+0000 7fd4337fe640 1 -- 172.21.15.5:0/1359234645 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd428001860 con 0x7fd4341457f0 2024-03-29T10:56:59.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.693+0000 7fd4337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:59.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.693+0000 7fd4337fe640 1 -- 172.21.15.5:0/1359234645 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fd428001c70 con 0x7fd4341457f0 2024-03-29T10:56:59.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.693+0000 7fd4337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:59.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.693+0000 7fd4337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:59.684 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.693+0000 7fd4337fe640 10 monclient: dump: 2024-03-29T10:56:59.684 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:59.684 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:59.684 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:59.685 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:59.685 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:59.685 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:59.685 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:59.685 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:59.685 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:59.685 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:59.686 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.693+0000 7fd43aab3640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:59.686 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.693+0000 7fd43aab3640 4 set_mon_vals no callback set 2024-03-29T10:56:59.686 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.694+0000 7fd43aab3640 10 monclient: shutdown 2024-03-29T10:56:59.686 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.694+0000 7fd43aab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:59.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.694+0000 7fd43aab3640 1 -- 172.21.15.5:0/1359234645 >> v1:172.21.15.5:6790/0 conn(0x7fd4341457f0 legacy=0x7fd43413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:59.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.694+0000 7fd43aab3640 1 -- 172.21.15.5:0/1359234645 shutdown_connections 2024-03-29T10:56:59.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.694+0000 7fd43aab3640 1 -- 172.21.15.5:0/1359234645 >> 172.21.15.5:0/1359234645 conn(0x7fd434066a00 msgr2=0x7fd434066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:59.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.695+0000 7fd43aab3640 1 -- 172.21.15.5:0/1359234645 shutdown_connections 2024-03-29T10:56:59.687 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.695+0000 7fd43aab3640 1 -- 172.21.15.5:0/1359234645 wait complete. 2024-03-29T10:56:59.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.696+0000 7fd43aab3640 10 monclient: build_initial_monmap 2024-03-29T10:56:59.688 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.696+0000 7fd43aab3640 10 monclient: monmap: 2024-03-29T10:56:59.688 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:56:59.688 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:59.689 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:56:59.689 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:56:59.689 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:59.689 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:59.689 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:56:59.689 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:56:59.689 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:56:59.689 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:59.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.696+0000 7fd43aab3640 1 librados: starting msgr at 2024-03-29T10:56:59.689 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.696+0000 7fd43aab3640 1 librados: starting objecter 2024-03-29T10:56:59.690 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.696+0000 7fd43aab3640 1 Processor -- start 2024-03-29T10:56:59.690 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.697+0000 7fd43aab3640 1 -- start start 2024-03-29T10:56:59.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.697+0000 7fd43aab3640 1 librados: setting wanted keys 2024-03-29T10:56:59.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.697+0000 7fd43aab3640 1 librados: calling monclient init 2024-03-29T10:56:59.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.697+0000 7fd43aab3640 10 monclient: init 2024-03-29T10:56:59.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.698+0000 7fd43aab3640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:59.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.698+0000 7fd43aab3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:59.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.698+0000 7fd43aab3640 10 monclient(hunting): picked mon.noname-c con 0x7fd4340a9d70 addr v1:172.21.15.5:6790/0 2024-03-29T10:56:59.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.698+0000 7fd43aab3640 10 monclient(hunting): picked mon.noname-a con 0x7fd43413f650 addr v1:172.21.15.5:6789/0 2024-03-29T10:56:59.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.698+0000 7fd43aab3640 10 monclient(hunting): picked mon.noname-b con 0x7fd4341457f0 addr v1:172.21.15.177:6789/0 2024-03-29T10:56:59.695 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.698+0000 7fd43aab3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4341deac0 con 0x7fd43413f650 2024-03-29T10:56:59.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.698+0000 7fd43aab3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd434141b40 con 0x7fd4341457f0 2024-03-29T10:56:59.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.698+0000 7fd43aab3640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd434144db0 con 0x7fd4340a9d70 2024-03-29T10:56:59.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.699+0000 7fd43884f640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fd4340a9d70 0x7fd4341df010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:58926/0 (socket says 172.21.15.5:58926) 2024-03-29T10:56:59.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.699+0000 7fd43884f640 1 -- 172.21.15.5:0/778819242 learned_addr learned my addr 172.21.15.5:0/778819242 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:56:59.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.700+0000 7fd43aab3640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:59.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.700+0000 7fd43aab3640 10 monclient(hunting): authenticate will time out at 1432.941348s 2024-03-29T10:56:59.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.702+0000 7fd4317fa640 10 client.?.objecter ms_handle_connect 0x7fd4340a9d70 2024-03-29T10:56:59.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.702+0000 7fd4317fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:59.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.702+0000 7fd4317fa640 10 client.?.objecter ms_handle_connect 0x7fd43413f650 2024-03-29T10:56:59.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.702+0000 7fd4317fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:59.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.702+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191704603 0 0) 0x7fd4341deac0 con 0x7fd43413f650 2024-03-29T10:56:59.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.702+0000 7fd4317fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:59.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.702+0000 7fd4317fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:59.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.702+0000 7fd4317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:59.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.702+0000 7fd4317fa640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:56:59.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd418004eb0 con 0x7fd43413f650 2024-03-29T10:56:59.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983570875 0 0) 0x7fd434144db0 con 0x7fd4340a9d70 2024-03-29T10:56:59.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:59.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:59.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:59.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): my global_id is 4631 2024-03-29T10:56:59.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4341deac0 con 0x7fd4340a9d70 2024-03-29T10:56:59.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3596573399 0 0) 0x7fd418004eb0 con 0x7fd43413f650 2024-03-29T10:56:59.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd434144db0 con 0x7fd43413f650 2024-03-29T10:56:59.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fd4200018b0 con 0x7fd43413f650 2024-03-29T10:56:59.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:59.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:59.699 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): dump: 2024-03-29T10:56:59.699 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:59.699 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:59.699 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:59.699 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:59.700 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:59.700 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:59.700 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:59.700 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:59.701 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:59.701 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:59.701 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1860626850 0 0) 0x7fd4341deac0 con 0x7fd4340a9d70 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd418004eb0 con 0x7fd4340a9d70 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fd428001c30 con 0x7fd4340a9d70 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.703+0000 7fd4317fa640 10 monclient(hunting): dump: 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:59.702 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.704+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1054436546 0 0) 0x7fd418004eb0 con 0x7fd4340a9d70 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.704+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 >> v1:172.21.15.177:6789/0 conn(0x7fd4341457f0 legacy=0x7fd4341e6e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 >> v1:172.21.15.5:6789/0 conn(0x7fd43413f650 legacy=0x7fd434142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd4317fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd4317fa640 1 monclient: found mon.c 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd4317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd4317fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd434145050 con 0x7fd4340a9d70 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd4317fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd4317fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd4317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:59.703 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd43aab3640 5 monclient: authenticate success, global_id 4631 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd43aab3640 10 monclient: _renew_subs 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd43aab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd43aab3640 1 -- 172.21.15.5:0/778819242 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd43413fda0 con 0x7fd4340a9d70 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd43aab3640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd43aab3640 10 monclient: _renew_subs 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd43aab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd43aab3640 1 -- 172.21.15.5:0/778819242 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4341df900 con 0x7fd4340a9d70 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.707+0000 7fd43aab3640 1 librados: init done 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.708+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd428004b00 con 0x7fd4340a9d70 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.708+0000 7fd4317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.708+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fd428004ed0 con 0x7fd4340a9d70 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.708+0000 7fd4317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.708+0000 7fd4317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:59.704 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.708+0000 7fd4317fa640 10 monclient: dump: 2024-03-29T10:56:59.705 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:56:59.705 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:56:59.705 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:59.705 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:56:59.705 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:59.705 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:56:59.705 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:56:59.705 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:56:59.705 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:56:59.705 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:56:59.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.708+0000 7fd432ffd640 4 set_mon_vals no callback set 2024-03-29T10:56:59.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.710+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fd428005030 con 0x7fd4340a9d70 2024-03-29T10:56:59.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.711+0000 7fd433fff640 10 monclient: get_auth_request con 0x7fd41806de30 auth_method 0 2024-03-29T10:56:59.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.711+0000 7fd43aab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:59.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.711+0000 7fd43aab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:59.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.711+0000 7fd43aab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:56:59.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.711+0000 7fd43aab3640 1 -- 172.21.15.5:0/778819242 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd414005080 con 0x7fd4340a9d70 2024-03-29T10:56:59.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.711+0000 7fd4317fa640 10 client.4631.objecter ms_handle_connect 0x7fd41806de30 2024-03-29T10:56:59.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.712+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fd428084ab0 con 0x7fd4340a9d70 2024-03-29T10:56:59.706 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.712+0000 7fd4317fa640 10 client.4631.objecter ms_dispatch 0x7fd434000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:59.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.712+0000 7fd4317fa640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:59.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.712+0000 7fd4317fa640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:59.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.712+0000 7fd4317fa640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T10:56:59.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.722+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd428050390 con 0x7fd4340a9d70 2024-03-29T10:56:59.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.722+0000 7fd4317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:59.707 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:56:59.722+0000 7fd4317fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:00.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.043+0000 7fd43aab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:00.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.044+0000 7fd43aab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:00.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.044+0000 7fd43aab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:00.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.044+0000 7fd43aab3640 1 -- 172.21.15.5:0/778819242 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd414005330 con 0x7fd4340a9d70 2024-03-29T10:57:00.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.044+0000 7fd4317fa640 1 -- 172.21.15.5:0/778819242 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fd4280522c0 con 0x7fd4340a9d70 2024-03-29T10:57:00.026 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:57:00.026 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:00.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.045+0000 7fd4317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:00.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.045+0000 7fd4317fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:00.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.048+0000 7fd43aab3640 10 librados: watch_flush enter 2024-03-29T10:57:00.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.048+0000 7fd43aab3640 10 librados: watch_flush exit 2024-03-29T10:57:00.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.048+0000 7fd43aab3640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T10:57:00.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.048+0000 7fd43aab3640 10 client.4631.objecter successfully canceled tick 2024-03-29T10:57:00.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.049+0000 7fd43aab3640 1 -- 172.21.15.5:0/778819242 >> v1:172.21.15.5:6816/2847500151 conn(0x7fd41806de30 legacy=0x7fd4180702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:00.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.049+0000 7fd43aab3640 10 monclient: shutdown 2024-03-29T10:57:00.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.049+0000 7fd43aab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:00.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.049+0000 7fd43aab3640 1 -- 172.21.15.5:0/778819242 >> v1:172.21.15.5:6790/0 conn(0x7fd4340a9d70 legacy=0x7fd4341df010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:00.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.050+0000 7fd43aab3640 1 -- 172.21.15.5:0/778819242 shutdown_connections 2024-03-29T10:57:00.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.050+0000 7fd43aab3640 1 -- 172.21.15.5:0/778819242 >> 172.21.15.5:0/778819242 conn(0x7fd434066a00 msgr2=0x7fd434066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:00.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.051+0000 7fd43aab3640 1 -- 172.21.15.5:0/778819242 shutdown_connections 2024-03-29T10:57:00.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.051+0000 7fd43aab3640 1 -- 172.21.15.5:0/778819242 wait complete. 2024-03-29T10:57:00.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:00.051+0000 7fd43aab3640 1 librados: shutdown 2024-03-29T10:57:00.049 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:06.856 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.018+0000 7fec04093640 10 monclient: get_monmap_and_config 2024-03-29T10:57:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.018+0000 7fec04093640 10 monclient: build_initial_monmap 2024-03-29T10:57:07.000 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.019+0000 7fec04093640 10 monclient: monmap: 2024-03-29T10:57:07.000 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:07.000 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:07.000 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:57:07.019985+0000 2024-03-29T10:57:07.000 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:57:07.019985+0000 2024-03-29T10:57:07.000 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:07.001 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:07.001 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:07.001 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:57:07.001 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:57:07.001 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:07.003 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.023+0000 7fec04093640 1 Processor -- start 2024-03-29T10:57:07.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.023+0000 7fec04093640 1 -- start start 2024-03-29T10:57:07.004 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.023+0000 7fec04093640 10 monclient: init 2024-03-29T10:57:07.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.024+0000 7fec04093640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:07.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.024+0000 7fec04093640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:07.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.024+0000 7fec04093640 10 monclient(hunting): picked mon.noname-c con 0x7febfc0b0770 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:07.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.024+0000 7fec04093640 10 monclient(hunting): picked mon.noname-a con 0x7febfc14a700 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:07.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.024+0000 7fec04093640 10 monclient(hunting): picked mon.noname-b con 0x7febfc0ac5a0 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:07.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.024+0000 7fec04093640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc0aba70 con 0x7febfc14a700 2024-03-29T10:57:07.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.024+0000 7fec04093640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc0aecb0 con 0x7febfc0b0770 2024-03-29T10:57:07.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.024+0000 7fec04093640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc0af150 con 0x7febfc0ac5a0 2024-03-29T10:57:07.006 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.024+0000 7fec04093640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:07.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.024+0000 7fec04093640 10 monclient(hunting): authenticate will time out at 1440.266051s 2024-03-29T10:57:07.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.025+0000 7fec02630640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7febfc0ac5a0 0x7febfc151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:40368/0 (socket says 172.21.15.5:40368) 2024-03-29T10:57:07.007 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.025+0000 7fec02630640 1 -- 172.21.15.5:0/1382495625 learned_addr learned my addr 172.21.15.5:0/1382495625 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.028+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456179990 0 0) 0x7febfc0af150 con 0x7febfc0ac5a0 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 10 monclient(hunting): _negotiate 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe8003520 con 0x7febfc0ac5a0 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551296329 0 0) 0x7febfc0aba70 con 0x7febfc14a700 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 10 monclient(hunting): _negotiate 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:57:07.013 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febfc0af150 con 0x7febfc14a700 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3418718442 0 0) 0x7febe8003520 con 0x7febfc0ac5a0 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 >> v1:172.21.15.177:6789/0 conn(0x7febfc0b0770 legacy=0x7febfc149ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.029+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 >> v1:172.21.15.5:6789/0 conn(0x7febfc14a700 legacy=0x7febfc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 10 monclient: _finish_hunting 0 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 1 monclient: found mon.noname-b 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc0ac9d0 con 0x7febfc0ac5a0 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 10 monclient: _finish_auth 0 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 10 monclient: _check_auth_tickets 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7febec001690 con 0x7febfc0ac5a0 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:07.014 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 10 monclient: dump: 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec04093640 5 monclient: authenticate success, global_id 4634 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec04093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febec001870 con 0x7febfc0ac5a0 2024-03-29T10:57:07.015 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec00e2d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec04093640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.030+0000 7fec04093640 4 set_mon_vals no callback set 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.031+0000 7fec00e2d640 1 -- 172.21.15.5:0/1382495625 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7febec001c80 con 0x7febfc0ac5a0 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.031+0000 7fec00e2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.031+0000 7fec00e2d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.031+0000 7fec00e2d640 10 monclient: dump: 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:07.016 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:07.017 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.031+0000 7fec04093640 10 monclient: shutdown 2024-03-29T10:57:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.031+0000 7fec04093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.031+0000 7fec04093640 1 -- 172.21.15.5:0/1382495625 >> v1:172.21.15.5:6790/0 conn(0x7febfc0ac5a0 legacy=0x7febfc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.032+0000 7fec04093640 1 -- 172.21.15.5:0/1382495625 shutdown_connections 2024-03-29T10:57:07.017 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.032+0000 7fec04093640 1 -- 172.21.15.5:0/1382495625 >> 172.21.15.5:0/1382495625 conn(0x7febfc06b660 msgr2=0x7febfc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.033+0000 7fec04093640 1 -- 172.21.15.5:0/1382495625 shutdown_connections 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.033+0000 7fec04093640 1 -- 172.21.15.5:0/1382495625 wait complete. 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.034+0000 7fec04093640 10 monclient: build_initial_monmap 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.034+0000 7fec04093640 10 monclient: monmap: 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:07.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 1 librados: starting msgr at 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 1 librados: starting objecter 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 1 Processor -- start 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 1 -- start start 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 1 librados: setting wanted keys 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 1 librados: calling monclient init 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 10 monclient: init 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 10 monclient(hunting): picked mon.noname-b con 0x7febfc0ac5a0 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.035+0000 7fec04093640 10 monclient(hunting): picked mon.noname-c con 0x7febfc0b0770 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.036+0000 7fec04093640 10 monclient(hunting): picked mon.noname-a con 0x7febfc14a700 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.036+0000 7fec04093640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc11f740 con 0x7febfc14a700 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.036+0000 7fec04093640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc1e8010 con 0x7febfc0ac5a0 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.036+0000 7fec04093640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc1e7c10 con 0x7febfc0b0770 2024-03-29T10:57:07.020 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.036+0000 7fec04093640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:07.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.036+0000 7fec04093640 10 monclient(hunting): authenticate will time out at 1440.277124s 2024-03-29T10:57:07.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.036+0000 7fec0162e640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7febfc0b0770 0x7febfc11d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:41232/0 (socket says 172.21.15.5:41232) 2024-03-29T10:57:07.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.036+0000 7fec0162e640 1 -- 172.21.15.5:0/3447346082 learned_addr learned my addr 172.21.15.5:0/3447346082 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:07.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.038+0000 7febf27fc640 10 client.?.objecter ms_handle_connect 0x7febfc0b0770 2024-03-29T10:57:07.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.038+0000 7febf27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:07.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.038+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935353092 0 0) 0x7febfc1e7c10 con 0x7febfc0b0770 2024-03-29T10:57:07.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:07.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:07.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:07.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 10 monclient(hunting): my global_id is 4637 2024-03-29T10:57:07.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe0004eb0 con 0x7febfc0b0770 2024-03-29T10:57:07.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1453040842 0 0) 0x7febe0004eb0 con 0x7febfc0b0770 2024-03-29T10:57:07.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febfc1e7c10 con 0x7febfc0b0770 2024-03-29T10:57:07.025 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7febf8001990 con 0x7febfc0b0770 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 10 monclient(hunting): dump: 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:07.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 974387218 0 0) 0x7febfc1e7c10 con 0x7febfc0b0770 2024-03-29T10:57:07.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 >> v1:172.21.15.177:6789/0 conn(0x7febfc0ac5a0 legacy=0x7febfc1e7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:07.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.039+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 >> v1:172.21.15.5:6789/0 conn(0x7febfc14a700 legacy=0x7febfc11dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:07.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:07.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 1 monclient: found mon.c 2024-03-29T10:57:07.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:07.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:07.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc1e7da0 con 0x7febfc0b0770 2024-03-29T10:57:07.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:07.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 10 client.?.objecter ms_handle_connect 0x7febfc14a700 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febf8001cd0 con 0x7febfc0b0770 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7febf8003a60 con 0x7febfc0b0770 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7febf27fc640 10 monclient: dump: 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:07.028 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7fec04093640 5 monclient: authenticate success, global_id 4637 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7fec04093640 10 monclient: _renew_subs 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7fec04093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.040+0000 7fec04093640 1 -- 172.21.15.5:0/3447346082 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febfc122ff0 con 0x7febfc0b0770 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.041+0000 7fec04093640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.041+0000 7fec04093640 10 monclient: _renew_subs 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.041+0000 7fec04093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:07.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.041+0000 7fec04093640 1 -- 172.21.15.5:0/3447346082 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febfc1efa30 con 0x7febfc0b0770 2024-03-29T10:57:07.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.041+0000 7fec04093640 1 librados: init done 2024-03-29T10:57:07.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.042+0000 7fec04093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:07.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.042+0000 7fec04093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:07.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.042+0000 7fec04093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:07.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.042+0000 7fec04093640 1 -- 172.21.15.5:0/3447346082 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febfc0ab560 con 0x7febfc0b0770 2024-03-29T10:57:07.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.042+0000 7febf3fff640 4 set_mon_vals no callback set 2024-03-29T10:57:07.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.046+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7febf8003bc0 con 0x7febfc0b0770 2024-03-29T10:57:07.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.046+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7febf8084ad0 con 0x7febfc0b0770 2024-03-29T10:57:07.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.046+0000 7febf27fc640 10 client.4637.objecter ms_dispatch 0x7febfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:07.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.046+0000 7febf27fc640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:07.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.046+0000 7febf27fc640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:07.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.046+0000 7febf27fc640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T10:57:07.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.047+0000 7fec01e2f640 10 monclient: get_auth_request con 0x7febe006dc20 auth_method 0 2024-03-29T10:57:07.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.047+0000 7febf27fc640 10 client.4637.objecter ms_handle_connect 0x7febe006dc20 2024-03-29T10:57:07.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.050+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febf80501e0 con 0x7febfc0b0770 2024-03-29T10:57:07.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.050+0000 7febf27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:07.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.050+0000 7febf27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:07.374 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:57:07.374 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:07.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.392+0000 7fec04093640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:07.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.392+0000 7fec04093640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:07.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.392+0000 7fec04093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:07.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.392+0000 7fec04093640 1 -- 172.21.15.5:0/3447346082 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febfc1e81a0 con 0x7febfc0b0770 2024-03-29T10:57:07.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.393+0000 7febf27fc640 1 -- 172.21.15.5:0/3447346082 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7febf80501e0 con 0x7febfc0b0770 2024-03-29T10:57:07.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.393+0000 7febf27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:07.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.393+0000 7febf27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:07.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.397+0000 7fec04093640 10 librados: watch_flush enter 2024-03-29T10:57:07.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.397+0000 7fec04093640 10 librados: watch_flush exit 2024-03-29T10:57:07.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.397+0000 7fec04093640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T10:57:07.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.397+0000 7fec04093640 10 client.4637.objecter successfully canceled tick 2024-03-29T10:57:07.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.397+0000 7fec04093640 1 -- 172.21.15.5:0/3447346082 >> v1:172.21.15.5:6816/2847500151 conn(0x7febe006dc20 legacy=0x7febe00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:07.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.397+0000 7fec04093640 10 monclient: shutdown 2024-03-29T10:57:07.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.397+0000 7fec04093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:07.378 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.397+0000 7fec04093640 1 -- 172.21.15.5:0/3447346082 >> v1:172.21.15.5:6790/0 conn(0x7febfc0b0770 legacy=0x7febfc11d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:07.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.399+0000 7fec04093640 1 -- 172.21.15.5:0/3447346082 shutdown_connections 2024-03-29T10:57:07.379 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.399+0000 7fec04093640 1 -- 172.21.15.5:0/3447346082 >> 172.21.15.5:0/3447346082 conn(0x7febfc06b660 msgr2=0x7febfc0abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:07.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.399+0000 7fec04093640 1 -- 172.21.15.5:0/3447346082 shutdown_connections 2024-03-29T10:57:07.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.399+0000 7fec04093640 1 -- 172.21.15.5:0/3447346082 wait complete. 2024-03-29T10:57:07.380 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:07.399+0000 7fec04093640 1 librados: shutdown 2024-03-29T10:57:07.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:14.203 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.372+0000 7f7b37d5c640 10 monclient: get_monmap_and_config 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.372+0000 7f7b37d5c640 10 monclient: build_initial_monmap 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.372+0000 7f7b37d5c640 10 monclient: monmap: 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:57:14.373213+0000 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:57:14.373213+0000 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:57:14.353 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:57:14.354 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:14.354 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.373+0000 7f7b37d5c640 1 Processor -- start 2024-03-29T10:57:14.354 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.373+0000 7f7b37d5c640 1 -- start start 2024-03-29T10:57:14.354 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.373+0000 7f7b37d5c640 10 monclient: init 2024-03-29T10:57:14.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.374+0000 7f7b37d5c640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:14.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.374+0000 7f7b37d5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:14.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.374+0000 7f7b37d5c640 10 monclient(hunting): picked mon.noname-b con 0x7f7b300b46b0 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:14.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.377+0000 7f7b37d5c640 10 monclient(hunting): picked mon.noname-a con 0x7f7b300aeae0 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:14.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.377+0000 7f7b37d5c640 10 monclient(hunting): picked mon.noname-c con 0x7f7b3013fa10 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:14.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.377+0000 7f7b37d5c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b300b06a0 con 0x7f7b300aeae0 2024-03-29T10:57:14.361 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.377+0000 7f7b37d5c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b300b4230 con 0x7f7b3013fa10 2024-03-29T10:57:14.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.377+0000 7f7b37d5c640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b300b0910 con 0x7f7b300b46b0 2024-03-29T10:57:14.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.377+0000 7f7b37d5c640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:14.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.377+0000 7f7b37d5c640 10 monclient(hunting): authenticate will time out at 1447.618685s 2024-03-29T10:57:14.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.380+0000 7f7b352f7640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7b300aeae0 0x7f7b3013f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:42527/0 (socket says 172.21.15.5:42527) 2024-03-29T10:57:14.362 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.380+0000 7f7b352f7640 1 -- 172.21.15.5:0/3974656912 learned_addr learned my addr 172.21.15.5:0/3974656912 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:14.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.381+0000 7f7b34af6640 1 -- 172.21.15.5:0/3974656912 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948942158 0 0) 0x7f7b300b06a0 con 0x7f7b300aeae0 2024-03-29T10:57:14.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.381+0000 7f7b34af6640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.381+0000 7f7b34af6640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.381+0000 7f7b34af6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.381+0000 7f7b34af6640 10 monclient(hunting): my global_id is 4533 2024-03-29T10:57:14.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.381+0000 7f7b34af6640 1 -- 172.21.15.5:0/3974656912 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b1c003520 con 0x7f7b300aeae0 2024-03-29T10:57:14.367 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.381+0000 7f7b34af6640 1 -- 172.21.15.5:0/3974656912 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2152185147 0 0) 0x7f7b1c003520 con 0x7f7b300aeae0 2024-03-29T10:57:14.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.381+0000 7f7b34af6640 1 -- 172.21.15.5:0/3974656912 >> v1:172.21.15.5:6790/0 conn(0x7f7b300b46b0 legacy=0x7f7b300b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:14.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.381+0000 7f7b34af6640 1 -- 172.21.15.5:0/3974656912 >> v1:172.21.15.177:6789/0 conn(0x7f7b3013fa10 legacy=0x7f7b3014d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:14.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.382+0000 7f7b34af6640 10 monclient: _finish_hunting 0 2024-03-29T10:57:14.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.382+0000 7f7b34af6640 1 monclient: found mon.noname-a 2024-03-29T10:57:14.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.382+0000 7f7b34af6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:14.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.382+0000 7f7b34af6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:57:14.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.382+0000 7f7b34af6640 1 -- 172.21.15.5:0/3974656912 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300af6d0 con 0x7f7b300aeae0 2024-03-29T10:57:14.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.382+0000 7f7b34af6640 10 monclient: _finish_auth 0 2024-03-29T10:57:14.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b34af6640 10 monclient: _check_auth_tickets 2024-03-29T10:57:14.368 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b34af6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b34af6640 1 -- 172.21.15.5:0/3974656912 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7b20001690 con 0x7f7b300aeae0 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b37d5c640 5 monclient: authenticate success, global_id 4533 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b37d5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b34af6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b34af6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b34af6640 10 monclient: dump: 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.369 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b34af6640 1 -- 172.21.15.5:0/3974656912 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b20001860 con 0x7f7b300aeae0 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b37d5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.383+0000 7f7b34af6640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.384+0000 7f7b34af6640 1 -- 172.21.15.5:0/3974656912 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7b20001cb0 con 0x7f7b300aeae0 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.384+0000 7f7b37d5c640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.384+0000 7f7b37d5c640 4 set_mon_vals no callback set 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.384+0000 7f7b34af6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.370 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.384+0000 7f7b34af6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:14.371 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.384+0000 7f7b34af6640 10 monclient: dump: 2024-03-29T10:57:14.371 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:14.371 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:14.371 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:14.371 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:14.371 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.371 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:14.372 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:14.372 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:14.372 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:14.372 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:14.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.385+0000 7f7b37d5c640 10 monclient: shutdown 2024-03-29T10:57:14.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.385+0000 7f7b37d5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:14.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.385+0000 7f7b37d5c640 1 -- 172.21.15.5:0/3974656912 >> v1:172.21.15.5:6789/0 conn(0x7f7b300aeae0 legacy=0x7f7b3013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.386+0000 7f7b37d5c640 1 -- 172.21.15.5:0/3974656912 shutdown_connections 2024-03-29T10:57:14.372 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.386+0000 7f7b37d5c640 1 -- 172.21.15.5:0/3974656912 >> 172.21.15.5:0/3974656912 conn(0x7f7b30066a00 msgr2=0x7f7b30066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:14.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.388+0000 7f7b37d5c640 1 -- 172.21.15.5:0/3974656912 shutdown_connections 2024-03-29T10:57:14.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.389+0000 7f7b37d5c640 1 -- 172.21.15.5:0/3974656912 wait complete. 2024-03-29T10:57:14.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.390+0000 7f7b37d5c640 10 monclient: build_initial_monmap 2024-03-29T10:57:14.374 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.390+0000 7f7b37d5c640 10 monclient: monmap: 2024-03-29T10:57:14.374 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:14.374 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:14.374 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:57:14.375 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:57:14.375 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:14.375 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:14.375 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:14.375 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:57:14.375 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:57:14.375 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:14.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.390+0000 7f7b37d5c640 1 librados: starting msgr at 2024-03-29T10:57:14.375 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.390+0000 7f7b37d5c640 1 librados: starting objecter 2024-03-29T10:57:14.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.391+0000 7f7b37d5c640 1 Processor -- start 2024-03-29T10:57:14.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.394+0000 7f7b37d5c640 1 -- start start 2024-03-29T10:57:14.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.394+0000 7f7b37d5c640 1 librados: setting wanted keys 2024-03-29T10:57:14.376 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.394+0000 7f7b37d5c640 1 librados: calling monclient init 2024-03-29T10:57:14.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.394+0000 7f7b37d5c640 10 monclient: init 2024-03-29T10:57:14.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.394+0000 7f7b37d5c640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:14.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.394+0000 7f7b37d5c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:14.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.394+0000 7f7b37d5c640 10 monclient(hunting): picked mon.noname-b con 0x7f7b300b46b0 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:14.377 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.394+0000 7f7b37d5c640 10 monclient(hunting): picked mon.noname-c con 0x7f7b3013fa10 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:14.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.397+0000 7f7b37d5c640 10 monclient(hunting): picked mon.noname-a con 0x7f7b3013fee0 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:14.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.397+0000 7f7b37d5c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b300b71e0 con 0x7f7b3013fee0 2024-03-29T10:57:14.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.397+0000 7f7b37d5c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b30141010 con 0x7f7b300b46b0 2024-03-29T10:57:14.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.397+0000 7f7b37d5c640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b301413c0 con 0x7f7b3013fa10 2024-03-29T10:57:14.384 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.397+0000 7f7b37d5c640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.397+0000 7f7b37d5c640 10 monclient(hunting): authenticate will time out at 1447.638638s 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.398+0000 7f7b362f9640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7b3013fee0 0x7f7b301e6600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44917/0 (socket says 172.21.15.5:44917) 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.398+0000 7f7b362f9640 1 -- 172.21.15.5:0/633115299 learned_addr learned my addr 172.21.15.5:0/633115299 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.398+0000 7f7b267fc640 10 client.?.objecter ms_handle_connect 0x7f7b3013fee0 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.398+0000 7f7b267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.398+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395104107 0 0) 0x7f7b300b71e0 con 0x7f7b3013fee0 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.399+0000 7f7b267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.399+0000 7f7b267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.399+0000 7f7b267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.399+0000 7f7b267fc640 10 monclient(hunting): my global_id is 4536 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.399+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b14004eb0 con 0x7f7b3013fee0 2024-03-29T10:57:14.385 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.399+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1113018390 0 0) 0x7f7b14004eb0 con 0x7f7b3013fee0 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.399+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b300b71e0 con 0x7f7b3013fee0 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.399+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7b28001b70 con 0x7f7b3013fee0 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.399+0000 7f7b267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 10 monclient(hunting): dump: 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:14.386 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 10 client.?.objecter ms_handle_connect 0x7f7b300b46b0 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228382774 0 0) 0x7f7b2c001090 con 0x7f7b300b46b0 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:57:14.387 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.400+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b14004eb0 con 0x7f7b300b46b0 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1992845911 0 0) 0x7f7b300b71e0 con 0x7f7b3013fee0 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 >> v1:172.21.15.5:6790/0 conn(0x7f7b3013fa10 legacy=0x7f7b300b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 >> v1:172.21.15.177:6789/0 conn(0x7f7b300b46b0 legacy=0x7f7b30145e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 1 monclient: found mon.a 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b30134790 con 0x7f7b3013fee0 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:14.388 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b37d5c640 5 monclient: authenticate success, global_id 4536 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b37d5c640 10 monclient: _renew_subs 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.401+0000 7f7b37d5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b37d5c640 1 -- 172.21.15.5:0/633115299 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b30141be0 con 0x7f7b3013fee0 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b28001cd0 con 0x7f7b3013fee0 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b37d5c640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b37d5c640 10 monclient: _renew_subs 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b37d5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b37d5c640 1 -- 172.21.15.5:0/633115299 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b301439b0 con 0x7f7b3013fee0 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b37d5c640 1 librados: init done 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b27fff640 4 set_mon_vals no callback set 2024-03-29T10:57:14.389 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f7b28004e20 con 0x7f7b3013fee0 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.402+0000 7f7b267fc640 10 monclient: dump: 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:14.390 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:14.391 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.404+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f7b28004f80 con 0x7f7b3013fee0 2024-03-29T10:57:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.405+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f7b28084e30 con 0x7f7b3013fee0 2024-03-29T10:57:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.405+0000 7f7b267fc640 10 client.4536.objecter ms_dispatch 0x7f7b30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.405+0000 7f7b267fc640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.405+0000 7f7b267fc640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.405+0000 7f7b267fc640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-29T10:57:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.405+0000 7f7b35af8640 10 monclient: get_auth_request con 0x7f7b1406db80 auth_method 0 2024-03-29T10:57:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.405+0000 7f7b267fc640 10 client.4536.objecter ms_handle_connect 0x7f7b1406db80 2024-03-29T10:57:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.409+0000 7f7b37d5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:14.392 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.409+0000 7f7b37d5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:14.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.409+0000 7f7b37d5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:14.393 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.413+0000 7f7b37d5c640 1 -- 172.21.15.5:0/633115299 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b18005080 con 0x7f7b3013fee0 2024-03-29T10:57:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.422+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b28050540 con 0x7f7b3013fee0 2024-03-29T10:57:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.422+0000 7f7b267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:14.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.422+0000 7f7b267fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:14.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.761+0000 7f7b37d5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:14.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.761+0000 7f7b37d5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:14.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.761+0000 7f7b37d5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:14.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.761+0000 7f7b37d5c640 1 -- 172.21.15.5:0/633115299 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b18005330 con 0x7f7b3013fee0 2024-03-29T10:57:14.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.762+0000 7f7b267fc640 1 -- 172.21.15.5:0/633115299 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f7b28052640 con 0x7f7b3013fee0 2024-03-29T10:57:14.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.762+0000 7f7b267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:14.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.762+0000 7f7b267fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:14.743 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:57:14.743 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:14.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.766+0000 7f7b37d5c640 10 librados: watch_flush enter 2024-03-29T10:57:14.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.766+0000 7f7b37d5c640 10 librados: watch_flush exit 2024-03-29T10:57:14.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.766+0000 7f7b37d5c640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-29T10:57:14.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.766+0000 7f7b37d5c640 10 client.4536.objecter successfully canceled tick 2024-03-29T10:57:14.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.766+0000 7f7b37d5c640 1 -- 172.21.15.5:0/633115299 >> v1:172.21.15.5:6816/2847500151 conn(0x7f7b1406db80 legacy=0x7f7b14070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.766+0000 7f7b37d5c640 10 monclient: shutdown 2024-03-29T10:57:14.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.766+0000 7f7b37d5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:14.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.766+0000 7f7b37d5c640 1 -- 172.21.15.5:0/633115299 >> v1:172.21.15.5:6789/0 conn(0x7f7b3013fee0 legacy=0x7f7b301e6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.767+0000 7f7b37d5c640 1 -- 172.21.15.5:0/633115299 shutdown_connections 2024-03-29T10:57:14.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.767+0000 7f7b37d5c640 1 -- 172.21.15.5:0/633115299 >> 172.21.15.5:0/633115299 conn(0x7f7b30066a00 msgr2=0x7f7b300b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:14.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.767+0000 7f7b37d5c640 1 -- 172.21.15.5:0/633115299 shutdown_connections 2024-03-29T10:57:14.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.767+0000 7f7b37d5c640 1 -- 172.21.15.5:0/633115299 wait complete. 2024-03-29T10:57:14.748 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:14.767+0000 7f7b37d5c640 1 librados: shutdown 2024-03-29T10:57:14.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:21.569 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:21.729 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.749+0000 7f0a0af4d640 10 monclient: get_monmap_and_config 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.749+0000 7f0a0af4d640 10 monclient: build_initial_monmap 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.749+0000 7f0a0af4d640 10 monclient: monmap: 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:57:21.750264+0000 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:57:21.750264+0000 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:57:21.730 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:57:21.731 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:21.731 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.750+0000 7f0a0af4d640 1 Processor -- start 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.750+0000 7f0a0af4d640 1 -- start start 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.750+0000 7f0a0af4d640 10 monclient: init 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.751+0000 7f0a0af4d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.751+0000 7f0a0af4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.751+0000 7f0a0af4d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a040b0770 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.751+0000 7f0a0af4d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a0414a700 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.751+0000 7f0a0af4d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a040ac5a0 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.751+0000 7f0a0af4d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a040aba70 con 0x7f0a0414a700 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.751+0000 7f0a0af4d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a040aecb0 con 0x7f0a040b0770 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.751+0000 7f0a0af4d640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a040af150 con 0x7f0a040ac5a0 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.751+0000 7f0a0af4d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:21.733 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.751+0000 7f0a0af4d640 10 monclient(hunting): authenticate will time out at 1454.992777s 2024-03-29T10:57:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.752+0000 7f0a094ea640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f0a040ac5a0 0x7f0a04151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:48304/0 (socket says 172.21.15.5:48304) 2024-03-29T10:57:21.735 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.752+0000 7f0a094ea640 1 -- 172.21.15.5:0/938208271 learned_addr learned my addr 172.21.15.5:0/938208271 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.755+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523887653 0 0) 0x7f0a040aba70 con 0x7f0a0414a700 2024-03-29T10:57:21.736 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.756+0000 7f0a037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.756+0000 7f0a037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.756+0000 7f0a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.756+0000 7f0a037fe640 10 monclient(hunting): my global_id is 4539 2024-03-29T10:57:21.737 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.756+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f4003520 con 0x7f0a0414a700 2024-03-29T10:57:21.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.756+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052210162 0 0) 0x7f0a040af150 con 0x7f0a040ac5a0 2024-03-29T10:57:21.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.756+0000 7f0a037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.757+0000 7f0a037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.757+0000 7f0a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.757+0000 7f0a037fe640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:57:21.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.757+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a040aba70 con 0x7f0a040ac5a0 2024-03-29T10:57:21.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.757+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955791905 0 0) 0x7f0a040aecb0 con 0x7f0a040b0770 2024-03-29T10:57:21.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.757+0000 7f0a037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.740 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.757+0000 7f0a037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.757+0000 7f0a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.757+0000 7f0a037fe640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.757+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a040af150 con 0x7f0a040b0770 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.758+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2167805362 0 0) 0x7f09f4003520 con 0x7f0a0414a700 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.758+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 >> v1:172.21.15.5:6790/0 conn(0x7f0a040ac5a0 legacy=0x7f0a04151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.758+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 >> v1:172.21.15.177:6789/0 conn(0x7f0a040b0770 legacy=0x7f0a04149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.758+0000 7f0a037fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.758+0000 7f0a037fe640 1 monclient: found mon.noname-a 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.758+0000 7f0a037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.758+0000 7f0a037fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.758+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a040ac9d0 con 0x7f0a0414a700 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.758+0000 7f0a037fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.759+0000 7f0a037fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.759+0000 7f0a037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:21.741 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.759+0000 7f0a0af4d640 5 monclient: authenticate success, global_id 4539 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.759+0000 7f0a0af4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.759+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f09f8001610 con 0x7f0a0414a700 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.759+0000 7f0a037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.759+0000 7f0a037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.759+0000 7f0a037fe640 10 monclient: dump: 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:21.742 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:21.743 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:21.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.760+0000 7f0a0af4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:21.743 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.760+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09f8001830 con 0x7f0a0414a700 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.760+0000 7f0a037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.760+0000 7f0a0af4d640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.760+0000 7f0a0af4d640 4 set_mon_vals no callback set 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.760+0000 7f0a037fe640 1 -- 172.21.15.5:0/938208271 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f09f8001bc0 con 0x7f0a0414a700 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.760+0000 7f0a037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.761+0000 7f0a037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.761+0000 7f0a037fe640 10 monclient: dump: 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.744 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:21.745 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:21.745 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:21.745 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:21.745 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:21.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.761+0000 7f0a0af4d640 10 monclient: shutdown 2024-03-29T10:57:21.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.761+0000 7f0a0af4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:21.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.761+0000 7f0a0af4d640 1 -- 172.21.15.5:0/938208271 >> v1:172.21.15.5:6789/0 conn(0x7f0a0414a700 legacy=0x7f0a040abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.764+0000 7f0a0af4d640 1 -- 172.21.15.5:0/938208271 shutdown_connections 2024-03-29T10:57:21.745 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.764+0000 7f0a0af4d640 1 -- 172.21.15.5:0/938208271 >> 172.21.15.5:0/938208271 conn(0x7f0a0406b660 msgr2=0x7f0a0406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:21.746 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.766+0000 7f0a0af4d640 1 -- 172.21.15.5:0/938208271 shutdown_connections 2024-03-29T10:57:21.747 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.766+0000 7f0a0af4d640 1 -- 172.21.15.5:0/938208271 wait complete. 2024-03-29T10:57:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.767+0000 7f0a0af4d640 10 monclient: build_initial_monmap 2024-03-29T10:57:21.749 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.767+0000 7f0a0af4d640 10 monclient: monmap: 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.767+0000 7f0a0af4d640 1 librados: starting msgr at 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.767+0000 7f0a0af4d640 1 librados: starting objecter 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.767+0000 7f0a0af4d640 1 Processor -- start 2024-03-29T10:57:21.750 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.768+0000 7f0a0af4d640 1 -- start start 2024-03-29T10:57:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.768+0000 7f0a0af4d640 1 librados: setting wanted keys 2024-03-29T10:57:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.768+0000 7f0a0af4d640 1 librados: calling monclient init 2024-03-29T10:57:21.751 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.768+0000 7f0a0af4d640 10 monclient: init 2024-03-29T10:57:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.769+0000 7f0a0af4d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.769+0000 7f0a0af4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.770+0000 7f0a0af4d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a040ac5a0 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.770+0000 7f0a0af4d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a040b0770 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.770+0000 7f0a0af4d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a0414a700 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.770+0000 7f0a0af4d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a041dbbc0 con 0x7f0a0414a700 2024-03-29T10:57:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.770+0000 7f0a0af4d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a04147930 con 0x7f0a040ac5a0 2024-03-29T10:57:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.770+0000 7f0a0af4d640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a041dc0e0 con 0x7f0a040b0770 2024-03-29T10:57:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.770+0000 7f0a0af4d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:21.752 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.770+0000 7f0a0af4d640 10 monclient(hunting): authenticate will time out at 1455.011743s 2024-03-29T10:57:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.774+0000 7f0a03fff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f0a040b0770 0x7f0a041de860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:51620/0 (socket says 172.21.15.5:51620) 2024-03-29T10:57:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.774+0000 7f0a03fff640 1 -- 172.21.15.5:0/1272133516 learned_addr learned my addr 172.21.15.5:0/1272133516 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.774+0000 7f0a017fa640 10 client.?.objecter ms_handle_connect 0x7f0a040b0770 2024-03-29T10:57:21.754 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.774+0000 7f0a017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.774+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886399127 0 0) 0x7f0a041dc0e0 con 0x7f0a040b0770 2024-03-29T10:57:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.774+0000 7f0a017fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.774+0000 7f0a017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.774+0000 7f0a017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.774+0000 7f0a017fa640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:57:21.755 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.774+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e8004eb0 con 0x7f0a040b0770 2024-03-29T10:57:21.756 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1218104605 0 0) 0x7f09e8004eb0 con 0x7f0a040b0770 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a041dc0e0 con 0x7f0a040b0770 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f09f8001bc0 con 0x7f0a040b0770 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 10 monclient(hunting): dump: 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:21.757 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3316615367 0 0) 0x7f0a041dc0e0 con 0x7f0a040b0770 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 >> v1:172.21.15.177:6789/0 conn(0x7f0a040ac5a0 legacy=0x7f0a041db360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 >> v1:172.21.15.5:6789/0 conn(0x7f0a0414a700 legacy=0x7f0a041e5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 10 monclient: _finish_hunting 0 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 1 monclient: found mon.c 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.775+0000 7f0a017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a041def70 con 0x7f0a040b0770 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 10 monclient: _finish_auth 0 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 10 monclient: _check_auth_tickets 2024-03-29T10:57:21.758 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:21.759 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09f80038a0 con 0x7f0a040b0770 2024-03-29T10:57:21.759 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:21.759 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f09f8003cf0 con 0x7f0a040b0770 2024-03-29T10:57:21.759 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.759 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:21.759 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a017fa640 10 monclient: dump: 2024-03-29T10:57:21.759 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:21.759 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:21.759 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:21.759 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.776+0000 7f0a02ffd640 4 set_mon_vals no callback set 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.778+0000 7f0a0af4d640 5 monclient: authenticate success, global_id 4643 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.778+0000 7f0a0af4d640 10 monclient: _renew_subs 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.778+0000 7f0a0af4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.778+0000 7f0a0af4d640 1 -- 172.21.15.5:0/1272133516 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a041df1c0 con 0x7f0a040b0770 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.778+0000 7f0a0af4d640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.778+0000 7f0a0af4d640 10 monclient: _renew_subs 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.778+0000 7f0a0af4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:21.760 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.778+0000 7f0a0af4d640 1 -- 172.21.15.5:0/1272133516 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a04070a30 con 0x7f0a040b0770 2024-03-29T10:57:21.761 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.778+0000 7f0a0af4d640 1 librados: init done 2024-03-29T10:57:21.761 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.779+0000 7f09eeffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:21.761 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.779+0000 7f09eeffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:21.761 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.779+0000 7f09eeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:21.761 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.779+0000 7f09eeffd640 1 -- 172.21.15.5:0/1272133516 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a041dbdc0 con 0x7f0a040b0770 2024-03-29T10:57:21.765 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.783+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f09f8003e50 con 0x7f0a040b0770 2024-03-29T10:57:21.765 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.784+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f09f8084b00 con 0x7f0a040b0770 2024-03-29T10:57:21.765 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.784+0000 7f0a017fa640 10 client.4643.objecter ms_dispatch 0x7f0a04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:21.765 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.784+0000 7f0a017fa640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:21.765 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.784+0000 7f0a017fa640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:21.765 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.784+0000 7f0a017fa640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T10:57:21.767 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.787+0000 7f0a08ce9640 10 monclient: get_auth_request con 0x7f09e806dc90 auth_method 0 2024-03-29T10:57:21.767 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.787+0000 7f0a017fa640 10 client.4643.objecter ms_handle_connect 0x7f09e806dc90 2024-03-29T10:57:21.771 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.790+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09f80503e0 con 0x7f0a040b0770 2024-03-29T10:57:21.771 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.790+0000 7f0a017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:21.771 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:21.790+0000 7f0a017fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:22.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.139+0000 7f09eeffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:22.122 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.139+0000 7f09eeffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:22.122 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.139+0000 7f09eeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:22.122 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.139+0000 7f09eeffd640 1 -- 172.21.15.5:0/1272133516 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a04127a10 con 0x7f0a040b0770 2024-03-29T10:57:22.123 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.142+0000 7f0a017fa640 1 -- 172.21.15.5:0/1272133516 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f09f8052310 con 0x7f0a040b0770 2024-03-29T10:57:22.123 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.142+0000 7f0a017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:22.123 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.142+0000 7f0a017fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:22.123 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:57:22.123 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:22.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.145+0000 7f09eeffd640 10 librados: watch_flush enter 2024-03-29T10:57:22.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.145+0000 7f09eeffd640 10 librados: watch_flush exit 2024-03-29T10:57:22.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.146+0000 7f09eeffd640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T10:57:22.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.146+0000 7f09eeffd640 10 client.4643.objecter successfully canceled tick 2024-03-29T10:57:22.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.146+0000 7f09eeffd640 1 -- 172.21.15.5:0/1272133516 >> v1:172.21.15.5:6816/2847500151 conn(0x7f09e806dc90 legacy=0x7f09e8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:22.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.146+0000 7f09eeffd640 10 monclient: shutdown 2024-03-29T10:57:22.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.146+0000 7f09eeffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:22.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.146+0000 7f09eeffd640 1 -- 172.21.15.5:0/1272133516 >> v1:172.21.15.5:6790/0 conn(0x7f0a040b0770 legacy=0x7f0a041de860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:22.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.146+0000 7f09eeffd640 1 -- 172.21.15.5:0/1272133516 shutdown_connections 2024-03-29T10:57:22.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.146+0000 7f09eeffd640 1 -- 172.21.15.5:0/1272133516 >> 172.21.15.5:0/1272133516 conn(0x7f0a0406b660 msgr2=0x7f0a040b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:22.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.147+0000 7f09eeffd640 1 -- 172.21.15.5:0/1272133516 shutdown_connections 2024-03-29T10:57:22.128 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.147+0000 7f09eeffd640 1 -- 172.21.15.5:0/1272133516 wait complete. 2024-03-29T10:57:22.128 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:22.147+0000 7f09eeffd640 1 librados: shutdown 2024-03-29T10:57:22.144 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:28.951 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:29.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.128+0000 7f0e0049a640 10 monclient: get_monmap_and_config 2024-03-29T10:57:29.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.128+0000 7f0e0049a640 10 monclient: build_initial_monmap 2024-03-29T10:57:29.111 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.128+0000 7f0e0049a640 10 monclient: monmap: 2024-03-29T10:57:29.111 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:29.111 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:29.111 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:57:29.129410+0000 2024-03-29T10:57:29.111 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:57:29.129410+0000 2024-03-29T10:57:29.111 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:29.111 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.129+0000 7f0e0049a640 1 Processor -- start 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.129+0000 7f0e0049a640 1 -- start start 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.129+0000 7f0e0049a640 10 monclient: init 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.130+0000 7f0e0049a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.130+0000 7f0e0049a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.130+0000 7f0e0049a640 10 monclient(hunting): picked mon.noname-c con 0x7f0df80af6c0 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:29.112 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.130+0000 7f0e0049a640 10 monclient(hunting): picked mon.noname-b con 0x7f0df814fb00 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:29.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.130+0000 7f0e0049a640 10 monclient(hunting): picked mon.noname-a con 0x7f0df8153090 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:29.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.130+0000 7f0e0049a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80a94e0 con 0x7f0df8153090 2024-03-29T10:57:29.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.131+0000 7f0e0049a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80afb60 con 0x7f0df80af6c0 2024-03-29T10:57:29.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.131+0000 7f0e0049a640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80aab80 con 0x7f0df814fb00 2024-03-29T10:57:29.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.131+0000 7f0e0049a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:29.113 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.131+0000 7f0e0049a640 10 monclient(hunting): authenticate will time out at 1462.372105s 2024-03-29T10:57:29.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.134+0000 7f0dfe236640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f0df80af6c0 0x7f0df80ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:35893/0 (socket says 172.21.15.5:35893) 2024-03-29T10:57:29.115 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.134+0000 7f0dfe236640 1 -- 172.21.15.5:0/485551972 learned_addr learned my addr 172.21.15.5:0/485551972 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:29.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 1 -- 172.21.15.5:0/485551972 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268855538 0 0) 0x7f0df80afb60 con 0x7f0df80af6c0 2024-03-29T10:57:29.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient(hunting): _negotiate 2024-03-29T10:57:29.118 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 1 -- 172.21.15.5:0/485551972 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de4003520 con 0x7f0df80af6c0 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 1 -- 172.21.15.5:0/485551972 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3051823736 0 0) 0x7f0de4003520 con 0x7f0df80af6c0 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 1 -- 172.21.15.5:0/485551972 >> v1:172.21.15.5:6790/0 conn(0x7f0df814fb00 legacy=0x7f0df8152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 1 -- 172.21.15.5:0/485551972 >> v1:172.21.15.5:6789/0 conn(0x7f0df8153090 legacy=0x7f0df815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient: _finish_hunting 0 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 1 monclient: found mon.noname-c 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 1 -- 172.21.15.5:0/485551972 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df80aa680 con 0x7f0df80af6c0 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient: _finish_auth 0 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient: _check_auth_tickets 2024-03-29T10:57:29.119 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0e0049a640 5 monclient: authenticate success, global_id 4486 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0e0049a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 1 -- 172.21.15.5:0/485551972 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f0df4001690 con 0x7f0df80af6c0 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0dfd234640 10 monclient: dump: 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:29.120 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.135+0000 7f0e0049a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.136+0000 7f0dfd234640 1 -- 172.21.15.5:0/485551972 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0df40018b0 con 0x7f0df80af6c0 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.136+0000 7f0dfd234640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.136+0000 7f0e0049a640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.136+0000 7f0e0049a640 4 set_mon_vals no callback set 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.137+0000 7f0e0049a640 10 monclient: shutdown 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.137+0000 7f0e0049a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.137+0000 7f0e0049a640 1 -- 172.21.15.5:0/485551972 >> v1:172.21.15.177:6789/0 conn(0x7f0df80af6c0 legacy=0x7f0df80ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.138+0000 7f0e0049a640 1 -- 172.21.15.5:0/485551972 shutdown_connections 2024-03-29T10:57:29.121 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.138+0000 7f0e0049a640 1 -- 172.21.15.5:0/485551972 >> 172.21.15.5:0/485551972 conn(0x7f0df80b35e0 msgr2=0x7f0df80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:29.122 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.140+0000 7f0e0049a640 1 -- 172.21.15.5:0/485551972 shutdown_connections 2024-03-29T10:57:29.122 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.140+0000 7f0e0049a640 1 -- 172.21.15.5:0/485551972 wait complete. 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.141+0000 7f0e0049a640 10 monclient: build_initial_monmap 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.141+0000 7f0e0049a640 10 monclient: monmap: 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:57:29.123 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:57:29.124 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:29.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.142+0000 7f0e0049a640 1 librados: starting msgr at 2024-03-29T10:57:29.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.142+0000 7f0e0049a640 1 librados: starting objecter 2024-03-29T10:57:29.124 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.142+0000 7f0e0049a640 1 Processor -- start 2024-03-29T10:57:29.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.144+0000 7f0e0049a640 1 -- start start 2024-03-29T10:57:29.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.144+0000 7f0e0049a640 1 librados: setting wanted keys 2024-03-29T10:57:29.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.144+0000 7f0e0049a640 1 librados: calling monclient init 2024-03-29T10:57:29.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.144+0000 7f0e0049a640 10 monclient: init 2024-03-29T10:57:29.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0e0049a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:29.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0e0049a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:29.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0e0049a640 10 monclient(hunting): picked mon.noname-a con 0x7f0df80af6c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:29.126 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0e0049a640 10 monclient(hunting): picked mon.noname-c con 0x7f0df814fb00 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:29.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0e0049a640 10 monclient(hunting): picked mon.noname-b con 0x7f0df8153090 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:29.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0e0049a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80713d0 con 0x7f0df80af6c0 2024-03-29T10:57:29.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0e0049a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df81e1d30 con 0x7f0df8153090 2024-03-29T10:57:29.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0e0049a640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df808af00 con 0x7f0df814fb00 2024-03-29T10:57:29.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0e0049a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:29.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0e0049a640 10 monclient(hunting): authenticate will time out at 1462.386457s 2024-03-29T10:57:29.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0dfda35640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f0df814fb00 0x7f0df808a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:49938/0 (socket says 172.21.15.5:49938) 2024-03-29T10:57:29.127 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.145+0000 7f0dfda35640 1 -- 172.21.15.5:0/1576860975 learned_addr learned my addr 172.21.15.5:0/1576860975 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.146+0000 7f0deeffd640 10 client.?.objecter ms_handle_connect 0x7f0df814fb00 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.146+0000 7f0deeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.146+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763466637 0 0) 0x7f0df808af00 con 0x7f0df814fb00 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.146+0000 7f0deeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.146+0000 7f0deeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.146+0000 7f0deeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.146+0000 7f0deeffd640 10 monclient(hunting): my global_id is 4649 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.146+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc004eb0 con 0x7f0df814fb00 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.147+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2137086504 0 0) 0x7f0ddc004eb0 con 0x7f0df814fb00 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.147+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0df808af00 con 0x7f0df814fb00 2024-03-29T10:57:29.129 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.147+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f0df0001d90 con 0x7f0df814fb00 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.147+0000 7f0deeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.148+0000 7f0deeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.148+0000 7f0deeffd640 10 monclient(hunting): dump: 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:29.130 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.148+0000 7f0deeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:29.131 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.148+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2860015131 0 0) 0x7f0df808af00 con 0x7f0df814fb00 2024-03-29T10:57:29.131 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.148+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 >> v1:172.21.15.177:6789/0 conn(0x7f0df8153090 legacy=0x7f0df81e44d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:57:29.131 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.148+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 >> v1:172.21.15.5:6789/0 conn(0x7f0df80af6c0 legacy=0x7f0df80718b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:29.131 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.149+0000 7f0deeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:29.131 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.149+0000 7f0deeffd640 1 monclient: found mon.c 2024-03-29T10:57:29.131 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.149+0000 7f0deeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.149+0000 7f0deeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.149+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df808b090 con 0x7f0df814fb00 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.149+0000 7f0deeffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.149+0000 7f0deeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.150+0000 7f0deeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.151+0000 7f0e0049a640 5 monclient: authenticate success, global_id 4649 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.151+0000 7f0e0049a640 10 monclient: _renew_subs 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.151+0000 7f0e0049a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.151+0000 7f0e0049a640 1 -- 172.21.15.5:0/1576860975 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df8070fc0 con 0x7f0df814fb00 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.151+0000 7f0e0049a640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.151+0000 7f0e0049a640 10 monclient: _renew_subs 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.151+0000 7f0e0049a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.151+0000 7f0e0049a640 1 -- 172.21.15.5:0/1576860975 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df80726f0 con 0x7f0df814fb00 2024-03-29T10:57:29.132 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.151+0000 7f0e0049a640 1 librados: init done 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.153+0000 7f0e0049a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.153+0000 7f0e0049a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.153+0000 7f0e0049a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.153+0000 7f0e0049a640 1 -- 172.21.15.5:0/1576860975 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df81e41f0 con 0x7f0df814fb00 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.154+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0df0001ef0 con 0x7f0df814fb00 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.154+0000 7f0deeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.154+0000 7f0deffff640 4 set_mon_vals no callback set 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.154+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f0df0003c40 con 0x7f0df814fb00 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.154+0000 7f0deeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.154+0000 7f0deeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.154+0000 7f0deeffd640 10 monclient: dump: 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:29.135 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:29.136 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:29.136 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:29.136 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:29.136 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:29.136 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:29.136 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:29.136 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:29.136 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:29.136 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.154+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f0df0003e60 con 0x7f0df814fb00 2024-03-29T10:57:29.137 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.155+0000 7f0dfe236640 10 monclient: get_auth_request con 0x7f0ddc06dc80 auth_method 0 2024-03-29T10:57:29.137 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.155+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7f0df0084ee0 con 0x7f0df814fb00 2024-03-29T10:57:29.137 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.155+0000 7f0deeffd640 10 client.4649.objecter ms_dispatch 0x7f0df8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:29.137 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.155+0000 7f0deeffd640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:29.137 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.155+0000 7f0deeffd640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:29.137 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.155+0000 7f0deeffd640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-29T10:57:29.137 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.156+0000 7f0deeffd640 10 client.4649.objecter ms_handle_connect 0x7f0ddc06dc80 2024-03-29T10:57:29.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.165+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0df81e41f0 con 0x7f0df814fb00 2024-03-29T10:57:29.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.165+0000 7f0deeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:29.146 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.165+0000 7f0deeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:29.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.530+0000 7f0e0049a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:29.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.530+0000 7f0e0049a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:29.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.530+0000 7f0e0049a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:29.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.530+0000 7f0e0049a640 1 -- 172.21.15.5:0/1576860975 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0df81e41f0 con 0x7f0df814fb00 2024-03-29T10:57:29.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.533+0000 7f0deeffd640 1 -- 172.21.15.5:0/1576860975 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7f0df81e41f0 con 0x7f0df814fb00 2024-03-29T10:57:29.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.533+0000 7f0deeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:29.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.533+0000 7f0deeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:29.515 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:57:29.515 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:29.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.538+0000 7f0e0049a640 10 librados: watch_flush enter 2024-03-29T10:57:29.518 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.538+0000 7f0e0049a640 10 librados: watch_flush exit 2024-03-29T10:57:29.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.538+0000 7f0e0049a640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-29T10:57:29.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.538+0000 7f0e0049a640 10 client.4649.objecter successfully canceled tick 2024-03-29T10:57:29.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.539+0000 7f0e0049a640 1 -- 172.21.15.5:0/1576860975 >> v1:172.21.15.5:6816/2847500151 conn(0x7f0ddc06dc80 legacy=0x7f0ddc070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:29.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.539+0000 7f0e0049a640 10 monclient: shutdown 2024-03-29T10:57:29.519 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.539+0000 7f0e0049a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:29.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.539+0000 7f0e0049a640 1 -- 172.21.15.5:0/1576860975 >> v1:172.21.15.5:6790/0 conn(0x7f0df814fb00 legacy=0x7f0df808a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:29.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.539+0000 7f0e0049a640 1 -- 172.21.15.5:0/1576860975 shutdown_connections 2024-03-29T10:57:29.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.539+0000 7f0e0049a640 1 -- 172.21.15.5:0/1576860975 >> 172.21.15.5:0/1576860975 conn(0x7f0df80b35e0 msgr2=0x7f0df80ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:29.520 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.540+0000 7f0e0049a640 1 -- 172.21.15.5:0/1576860975 shutdown_connections 2024-03-29T10:57:29.521 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.540+0000 7f0e0049a640 1 -- 172.21.15.5:0/1576860975 wait complete. 2024-03-29T10:57:29.521 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:29.540+0000 7f0e0049a640 1 librados: shutdown 2024-03-29T10:57:29.542 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:36.349 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:36.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.508+0000 7fa74d3a8640 10 monclient: get_monmap_and_config 2024-03-29T10:57:36.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.508+0000 7fa74d3a8640 10 monclient: build_initial_monmap 2024-03-29T10:57:36.492 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.508+0000 7fa74d3a8640 10 monclient: monmap: 2024-03-29T10:57:36.492 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:36.492 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:36.492 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:57:36.509065+0000 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:57:36.509065+0000 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.508+0000 7fa74d3a8640 1 Processor -- start 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.509+0000 7fa74d3a8640 1 -- start start 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.509+0000 7fa74d3a8640 10 monclient: init 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.509+0000 7fa74d3a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.509+0000 7fa74d3a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.510+0000 7fa74d3a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa74806c250 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.510+0000 7fa74d3a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa748072400 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:36.493 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.510+0000 7fa74d3a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa74806c630 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:36.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.510+0000 7fa74d3a8640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74806d760 con 0x7fa74806c250 2024-03-29T10:57:36.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.510+0000 7fa74d3a8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa748070500 con 0x7fa748072400 2024-03-29T10:57:36.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.510+0000 7fa74d3a8640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7480aff50 con 0x7fa74806c630 2024-03-29T10:57:36.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.510+0000 7fa74d3a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:36.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.510+0000 7fa74d3a8640 10 monclient(hunting): authenticate will time out at 1469.751159s 2024-03-29T10:57:36.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.510+0000 7fa7477fe640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fa74806c630 0x7fa748151ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:41902/0 (socket says 172.21.15.5:41902) 2024-03-29T10:57:36.494 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.510+0000 7fa7477fe640 1 -- 172.21.15.5:0/1946107365 learned_addr learned my addr 172.21.15.5:0/1946107365 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.512+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178879108 0 0) 0x7fa74806d760 con 0x7fa74806c250 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.512+0000 7fa745ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.512+0000 7fa745ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.512+0000 7fa745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.512+0000 7fa745ffb640 10 monclient(hunting): my global_id is 4545 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.512+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa730003520 con 0x7fa74806c250 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.513+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186472156 0 0) 0x7fa7480aff50 con 0x7fa74806c630 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.513+0000 7fa745ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.513+0000 7fa745ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.513+0000 7fa745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.513+0000 7fa745ffb640 10 monclient(hunting): my global_id is 4652 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.513+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa74806d760 con 0x7fa74806c630 2024-03-29T10:57:36.495 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.514+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3490196889 0 0) 0x7fa74806d760 con 0x7fa74806c630 2024-03-29T10:57:36.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.514+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 >> v1:172.21.15.177:6789/0 conn(0x7fa748072400 legacy=0x7fa7481465b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:36.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 >> v1:172.21.15.5:6789/0 conn(0x7fa74806c250 legacy=0x7fa748071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:36.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 1 monclient: found mon.noname-b 2024-03-29T10:57:36.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:36.496 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7480b00e0 con 0x7fa74806c630 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa7400019f0 con 0x7fa74806c630 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 10 monclient: dump: 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:36.497 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:36.498 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:36.498 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:36.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa74d3a8640 5 monclient: authenticate success, global_id 4652 2024-03-29T10:57:36.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa74d3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:36.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa740001b50 con 0x7fa74806c630 2024-03-29T10:57:36.498 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa745ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:36.499 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa74d3a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:36.499 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.515+0000 7fa74d3a8640 4 set_mon_vals no callback set 2024-03-29T10:57:36.499 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.516+0000 7fa745ffb640 1 -- 172.21.15.5:0/1946107365 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa740001f60 con 0x7fa74806c630 2024-03-29T10:57:36.499 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.516+0000 7fa745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:36.499 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.516+0000 7fa745ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:36.499 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.516+0000 7fa745ffb640 10 monclient: dump: 2024-03-29T10:57:36.499 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.516+0000 7fa74d3a8640 10 monclient: shutdown 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.516+0000 7fa74d3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.516+0000 7fa74d3a8640 1 -- 172.21.15.5:0/1946107365 >> v1:172.21.15.5:6790/0 conn(0x7fa74806c630 legacy=0x7fa748151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.516+0000 7fa74d3a8640 1 -- 172.21.15.5:0/1946107365 shutdown_connections 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.516+0000 7fa74d3a8640 1 -- 172.21.15.5:0/1946107365 >> 172.21.15.5:0/1946107365 conn(0x7fa748067610 msgr2=0x7fa7480b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:36.500 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.517+0000 7fa74d3a8640 1 -- 172.21.15.5:0/1946107365 shutdown_connections 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.517+0000 7fa74d3a8640 1 -- 172.21.15.5:0/1946107365 wait complete. 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.518+0000 7fa74d3a8640 10 monclient: build_initial_monmap 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.518+0000 7fa74d3a8640 10 monclient: monmap: 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.518+0000 7fa74d3a8640 1 librados: starting msgr at 2024-03-29T10:57:36.501 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.518+0000 7fa74d3a8640 1 librados: starting objecter 2024-03-29T10:57:36.502 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.519+0000 7fa74d3a8640 1 Processor -- start 2024-03-29T10:57:36.508 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.519+0000 7fa74d3a8640 1 -- start start 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.519+0000 7fa74d3a8640 1 librados: setting wanted keys 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.519+0000 7fa74d3a8640 1 librados: calling monclient init 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.519+0000 7fa74d3a8640 10 monclient: init 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa74d3a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa74d3a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa74d3a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa74806c250 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa74d3a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa74806c630 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa74d3a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa748072400 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa74d3a8640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7480b3910 con 0x7fa748072400 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa74d3a8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7480abfd0 con 0x7fa74806c250 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa74d3a8640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7480a9c50 con 0x7fa74806c630 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa74d3a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa74d3a8640 10 monclient(hunting): authenticate will time out at 1469.761498s 2024-03-29T10:57:36.509 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa7477fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa748072400 0x7fa7481e62e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36840/0 (socket says 172.21.15.5:36840) 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.520+0000 7fa7477fe640 1 -- 172.21.15.5:0/2202800309 learned_addr learned my addr 172.21.15.5:0/2202800309 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 client.?.objecter ms_handle_connect 0x7fa748072400 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 client.?.objecter ms_handle_connect 0x7fa74806c630 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877990069 0 0) 0x7fa7480a9c50 con 0x7fa74806c630 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): my global_id is 4655 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa734004eb0 con 0x7fa74806c630 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429620888 0 0) 0x7fa7480b3910 con 0x7fa748072400 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:36.510 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): my global_id is 4548 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7480a9c50 con 0x7fa748072400 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3805148664 0 0) 0x7fa734004eb0 con 0x7fa74806c630 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7480b3910 con 0x7fa74806c630 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa73c001700 con 0x7fa74806c630 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:36.511 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 738009651 0 0) 0x7fa7480b3910 con 0x7fa74806c630 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 >> v1:172.21.15.177:6789/0 conn(0x7fa74806c250 legacy=0x7fa7480b3df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 >> v1:172.21.15.5:6789/0 conn(0x7fa748072400 legacy=0x7fa7481e62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 1 monclient: found mon.c 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.521+0000 7fa72f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:36.512 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7480ab040 con 0x7fa74806c630 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa74d3a8640 5 monclient: authenticate success, global_id 4655 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa74d3a8640 10 monclient: _renew_subs 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa74d3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa74d3a8640 1 -- 172.21.15.5:0/2202800309 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7481dfa30 con 0x7fa74806c630 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa74d3a8640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa74d3a8640 10 monclient: _renew_subs 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa74d3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa74d3a8640 1 -- 172.21.15.5:0/2202800309 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7481df1f0 con 0x7fa74806c630 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa74d3a8640 1 librados: init done 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa73c001d50 con 0x7fa74806c630 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:36.513 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa73c003790 con 0x7fa74806c630 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa72f7fe640 10 monclient: dump: 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:36.514 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.522+0000 7fa7457fa640 4 set_mon_vals no callback set 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.524+0000 7fa74d3a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.524+0000 7fa74d3a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.524+0000 7fa74d3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.524+0000 7fa74d3a8640 1 -- 172.21.15.5:0/2202800309 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa728005080 con 0x7fa74806c630 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.524+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fa73c019760 con 0x7fa74806c630 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.525+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fa73c083750 con 0x7fa74806c630 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.525+0000 7fa746ffd640 10 monclient: get_auth_request con 0x7fa73406dd40 auth_method 0 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.525+0000 7fa72f7fe640 10 client.4655.objecter ms_dispatch 0x7fa748000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.525+0000 7fa72f7fe640 3 client.4655.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.525+0000 7fa72f7fe640 3 client.4655.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.525+0000 7fa72f7fe640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-29T10:57:36.515 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.526+0000 7fa72f7fe640 10 client.4655.objecter ms_handle_connect 0x7fa73406dd40 2024-03-29T10:57:36.516 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.531+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa73c04ee60 con 0x7fa74806c630 2024-03-29T10:57:36.516 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.531+0000 7fa72f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:36.516 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.531+0000 7fa72f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:36.839 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.859+0000 7fa74d3a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:36.839 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.859+0000 7fa74d3a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:36.840 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.859+0000 7fa74d3a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:57:36.840 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.859+0000 7fa74d3a8640 1 -- 172.21.15.5:0/2202800309 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa728005330 con 0x7fa74806c630 2024-03-29T10:57:36.840 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.860+0000 7fa72f7fe640 1 -- 172.21.15.5:0/2202800309 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fa73c050f60 con 0x7fa74806c630 2024-03-29T10:57:36.840 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.860+0000 7fa72f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:36.840 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.860+0000 7fa72f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:36.841 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:57:36.841 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:36.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.863+0000 7fa74d3a8640 10 librados: watch_flush enter 2024-03-29T10:57:36.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.864+0000 7fa74d3a8640 10 librados: watch_flush exit 2024-03-29T10:57:36.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.864+0000 7fa74d3a8640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-29T10:57:36.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.864+0000 7fa74d3a8640 10 client.4655.objecter successfully canceled tick 2024-03-29T10:57:36.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.864+0000 7fa74d3a8640 1 -- 172.21.15.5:0/2202800309 >> v1:172.21.15.5:6816/2847500151 conn(0x7fa73406dd40 legacy=0x7fa734070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.864+0000 7fa74d3a8640 10 monclient: shutdown 2024-03-29T10:57:36.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.864+0000 7fa74d3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:36.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.864+0000 7fa74d3a8640 1 -- 172.21.15.5:0/2202800309 >> v1:172.21.15.5:6790/0 conn(0x7fa74806c630 legacy=0x7fa7480b4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.864+0000 7fa74d3a8640 1 -- 172.21.15.5:0/2202800309 shutdown_connections 2024-03-29T10:57:36.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.864+0000 7fa74d3a8640 1 -- 172.21.15.5:0/2202800309 >> 172.21.15.5:0/2202800309 conn(0x7fa748067610 msgr2=0x7fa748149ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:36.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.864+0000 7fa74d3a8640 1 -- 172.21.15.5:0/2202800309 shutdown_connections 2024-03-29T10:57:36.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.865+0000 7fa74d3a8640 1 -- 172.21.15.5:0/2202800309 wait complete. 2024-03-29T10:57:36.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:36.865+0000 7fa74d3a8640 1 librados: shutdown 2024-03-29T10:57:36.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:43.671 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:43.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.845+0000 7fc35000f640 10 monclient: get_monmap_and_config 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.845+0000 7fc35000f640 10 monclient: build_initial_monmap 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.845+0000 7fc35000f640 10 monclient: monmap: 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:57:43.846041+0000 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:57:43.846041+0000 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:57:43.826 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:43.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.845+0000 7fc35000f640 1 Processor -- start 2024-03-29T10:57:43.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.846+0000 7fc35000f640 1 -- start start 2024-03-29T10:57:43.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.846+0000 7fc35000f640 10 monclient: init 2024-03-29T10:57:43.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.846+0000 7fc35000f640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.846+0000 7fc35000f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc35000f640 10 monclient(hunting): picked mon.noname-c con 0x7fc3480b0500 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc35000f640 10 monclient(hunting): picked mon.noname-b con 0x7fc3480b08e0 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc35000f640 10 monclient(hunting): picked mon.noname-a con 0x7fc3480aeca0 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc35000f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc348140620 con 0x7fc3480aeca0 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc35000f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc34813daf0 con 0x7fc3480b0500 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc35000f640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3480ae180 con 0x7fc3480b08e0 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc35000f640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc35000f640 10 monclient(hunting): authenticate will time out at 1477.088170s 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34d5aa640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fc3480b08e0 0x7fc34813d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:36378/0 (socket says 172.21.15.5:36378) 2024-03-29T10:57:43.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34d5aa640 1 -- 172.21.15.5:0/1640214095 learned_addr learned my addr 172.21.15.5:0/1640214095 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:43.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037361698 0 0) 0x7fc348140620 con 0x7fc3480aeca0 2024-03-29T10:57:43.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 10 monclient(hunting): my global_id is 4551 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc334003520 con 0x7fc3480aeca0 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606885887 0 0) 0x7fc3480ae180 con 0x7fc3480b08e0 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 10 monclient(hunting): my global_id is 4658 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.847+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc348140620 con 0x7fc3480b08e0 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3541057161 0 0) 0x7fc348140620 con 0x7fc3480b08e0 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 >> v1:172.21.15.177:6789/0 conn(0x7fc3480b0500 legacy=0x7fc34813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 >> v1:172.21.15.5:6789/0 conn(0x7fc3480aeca0 legacy=0x7fc34814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: _finish_hunting 0 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 1 monclient: found mon.noname-b 2024-03-29T10:57:43.832 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3480accf0 con 0x7fc3480b08e0 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: _finish_auth 0 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: _check_auth_tickets 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc338001670 con 0x7fc3480b08e0 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: dump: 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:43.833 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc35000f640 5 monclient: authenticate success, global_id 4658 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc35000f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3380017d0 con 0x7fc3480b08e0 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 1 -- 172.21.15.5:0/1640214095 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc338001c20 con 0x7fc3480b08e0 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc34cda9640 10 monclient: dump: 2024-03-29T10:57:43.834 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc35000f640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.848+0000 7fc35000f640 4 set_mon_vals no callback set 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.849+0000 7fc35000f640 10 monclient: shutdown 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.849+0000 7fc35000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.849+0000 7fc35000f640 1 -- 172.21.15.5:0/1640214095 >> v1:172.21.15.5:6790/0 conn(0x7fc3480b08e0 legacy=0x7fc34813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.835 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.849+0000 7fc35000f640 1 -- 172.21.15.5:0/1640214095 shutdown_connections 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.849+0000 7fc35000f640 1 -- 172.21.15.5:0/1640214095 >> 172.21.15.5:0/1640214095 conn(0x7fc34806b660 msgr2=0x7fc34806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.849+0000 7fc35000f640 1 -- 172.21.15.5:0/1640214095 shutdown_connections 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.849+0000 7fc35000f640 1 -- 172.21.15.5:0/1640214095 wait complete. 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.850+0000 7fc35000f640 10 monclient: build_initial_monmap 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.850+0000 7fc35000f640 10 monclient: monmap: 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:57:43.836 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:57:43.837 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:43.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.850+0000 7fc35000f640 1 librados: starting msgr at 2024-03-29T10:57:43.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.850+0000 7fc35000f640 1 librados: starting objecter 2024-03-29T10:57:43.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.850+0000 7fc35000f640 1 Processor -- start 2024-03-29T10:57:43.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 1 -- start start 2024-03-29T10:57:43.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 1 librados: setting wanted keys 2024-03-29T10:57:43.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 1 librados: calling monclient init 2024-03-29T10:57:43.837 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 10 monclient: init 2024-03-29T10:57:43.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:43.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:43.843 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 10 monclient(hunting): picked mon.noname-b con 0x7fc3480aeca0 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 10 monclient(hunting): picked mon.noname-c con 0x7fc3480b0500 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 10 monclient(hunting): picked mon.noname-a con 0x7fc34813adb0 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3481db5f0 con 0x7fc34813adb0 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3481db780 con 0x7fc3480aeca0 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc348126330 con 0x7fc3480b0500 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.851+0000 7fc35000f640 10 monclient(hunting): authenticate will time out at 1477.093009s 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.852+0000 7fc34d5aa640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fc3480b0500 0x7fc348125c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:36516/0 (socket says 172.21.15.5:36516) 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.852+0000 7fc34d5aa640 1 -- 172.21.15.5:0/927868813 learned_addr learned my addr 172.21.15.5:0/927868813 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.852+0000 7fc33e7fc640 10 client.?.objecter ms_handle_connect 0x7fc34813adb0 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.852+0000 7fc33e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.852+0000 7fc33e7fc640 10 client.?.objecter ms_handle_connect 0x7fc3480b0500 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.852+0000 7fc33e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:43.844 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 client.?.objecter ms_handle_connect 0x7fc3480aeca0 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804219852 0 0) 0x7fc3481db5f0 con 0x7fc34813adb0 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): my global_id is 4554 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc32c004eb0 con 0x7fc34813adb0 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898692182 0 0) 0x7fc348126330 con 0x7fc3480b0500 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): my global_id is 4661 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3481db5f0 con 0x7fc3480b0500 2024-03-29T10:57:43.845 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917473191 0 0) 0x7fc3481db780 con 0x7fc3480aeca0 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 10 monclient(hunting): my global_id is 4489 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc348126330 con 0x7fc3480aeca0 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3229891751 0 0) 0x7fc3481db5f0 con 0x7fc3480b0500 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.853+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3481db780 con 0x7fc3480b0500 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc338001b60 con 0x7fc3480b0500 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient(hunting): dump: 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:43.846 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 307013934 0 0) 0x7fc32c004eb0 con 0x7fc34813adb0 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3481db5f0 con 0x7fc34813adb0 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc340001c50 con 0x7fc34813adb0 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient(hunting): dump: 2024-03-29T10:57:43.847 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2412462463 0 0) 0x7fc3481db5f0 con 0x7fc34813adb0 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 >> v1:172.21.15.5:6790/0 conn(0x7fc3480b0500 legacy=0x7fc348125c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 >> v1:172.21.15.177:6789/0 conn(0x7fc3480aeca0 legacy=0x7fc34806fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 1 monclient: found mon.a 2024-03-29T10:57:43.848 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3481264c0 con 0x7fc34813adb0 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc33e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc35000f640 5 monclient: authenticate success, global_id 4554 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc35000f640 10 monclient: _renew_subs 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc35000f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc35000f640 1 -- 172.21.15.5:0/927868813 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc348125240 con 0x7fc34813adb0 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc35000f640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc35000f640 10 monclient: _renew_subs 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc35000f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc35000f640 1 -- 172.21.15.5:0/927868813 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3481e77c0 con 0x7fc34813adb0 2024-03-29T10:57:43.849 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.854+0000 7fc35000f640 1 librados: init done 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.855+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc340003990 con 0x7fc34813adb0 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.856+0000 7fc33e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.856+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fc340003ca0 con 0x7fc34813adb0 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.856+0000 7fc33e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.856+0000 7fc33ffff640 4 set_mon_vals no callback set 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.856+0000 7fc33e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.856+0000 7fc33e7fc640 10 monclient: dump: 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:43.850 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.856+0000 7fc35000f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.857+0000 7fc35000f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.857+0000 7fc35000f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.857+0000 7fc35000f640 1 -- 172.21.15.5:0/927868813 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc330005080 con 0x7fc34813adb0 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.860+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fc340019c70 con 0x7fc34813adb0 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.861+0000 7fc34ddab640 10 monclient: get_auth_request con 0x7fc32c06dff0 auth_method 0 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.861+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fc340083aa0 con 0x7fc34813adb0 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.861+0000 7fc33e7fc640 10 client.4554.objecter ms_dispatch 0x7fc348000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.861+0000 7fc33e7fc640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.862+0000 7fc33e7fc640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:43.851 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.862+0000 7fc33e7fc640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T10:57:43.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.862+0000 7fc33e7fc640 10 client.4554.objecter ms_handle_connect 0x7fc32c06dff0 2024-03-29T10:57:43.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.866+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc34004f380 con 0x7fc34813adb0 2024-03-29T10:57:43.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.867+0000 7fc33e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:43.852 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:43.867+0000 7fc33e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:44.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.264+0000 7fc35000f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:44.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.264+0000 7fc35000f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:44.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.264+0000 7fc35000f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:44.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.264+0000 7fc35000f640 1 -- 172.21.15.5:0/927868813 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc330005330 con 0x7fc34813adb0 2024-03-29T10:57:44.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.266+0000 7fc33e7fc640 1 -- 172.21.15.5:0/927868813 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fc3400512b0 con 0x7fc34813adb0 2024-03-29T10:57:44.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.266+0000 7fc33e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:44.247 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.266+0000 7fc33e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:44.247 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:57:44.247 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:44.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.270+0000 7fc35000f640 10 librados: watch_flush enter 2024-03-29T10:57:44.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.270+0000 7fc35000f640 10 librados: watch_flush exit 2024-03-29T10:57:44.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.270+0000 7fc35000f640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T10:57:44.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.270+0000 7fc35000f640 10 client.4554.objecter successfully canceled tick 2024-03-29T10:57:44.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.270+0000 7fc35000f640 1 -- 172.21.15.5:0/927868813 >> v1:172.21.15.5:6816/2847500151 conn(0x7fc32c06dff0 legacy=0x7fc32c0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:44.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.270+0000 7fc35000f640 10 monclient: shutdown 2024-03-29T10:57:44.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.270+0000 7fc35000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:44.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.270+0000 7fc35000f640 1 -- 172.21.15.5:0/927868813 >> v1:172.21.15.5:6789/0 conn(0x7fc34813adb0 legacy=0x7fc3481e60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:44.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.270+0000 7fc35000f640 1 -- 172.21.15.5:0/927868813 shutdown_connections 2024-03-29T10:57:44.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.270+0000 7fc35000f640 1 -- 172.21.15.5:0/927868813 >> 172.21.15.5:0/927868813 conn(0x7fc34806b660 msgr2=0x7fc3481400d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:44.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.271+0000 7fc35000f640 1 -- 172.21.15.5:0/927868813 shutdown_connections 2024-03-29T10:57:44.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.271+0000 7fc35000f640 1 -- 172.21.15.5:0/927868813 wait complete. 2024-03-29T10:57:44.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:44.271+0000 7fc35000f640 1 librados: shutdown 2024-03-29T10:57:44.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:51.071 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:51.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.246+0000 7fa8dffff640 10 monclient: get_monmap_and_config 2024-03-29T10:57:51.226 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.246+0000 7fa8dffff640 10 monclient: build_initial_monmap 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.246+0000 7fa8dffff640 10 monclient: monmap: 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:57:51.247126+0000 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:57:51.247126+0000 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:57:51.227 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:51.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.249+0000 7fa8dffff640 1 Processor -- start 2024-03-29T10:57:51.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.250+0000 7fa8dffff640 1 -- start start 2024-03-29T10:57:51.230 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.250+0000 7fa8dffff640 10 monclient: init 2024-03-29T10:57:51.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.251+0000 7fa8dffff640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:51.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.251+0000 7fa8dffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:51.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.251+0000 7fa8dffff640 10 monclient(hunting): picked mon.noname-c con 0x7fa8d80389e0 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:51.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.251+0000 7fa8dffff640 10 monclient(hunting): picked mon.noname-a con 0x7fa8d8038dc0 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:51.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.251+0000 7fa8dffff640 10 monclient(hunting): picked mon.noname-b con 0x7fa8d8030660 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:51.231 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.251+0000 7fa8dffff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d80c54c0 con 0x7fa8d8038dc0 2024-03-29T10:57:51.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.251+0000 7fa8dffff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d8036430 con 0x7fa8d80389e0 2024-03-29T10:57:51.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.251+0000 7fa8dffff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d8039840 con 0x7fa8d8030660 2024-03-29T10:57:51.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.251+0000 7fa8dffff640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:51.232 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.251+0000 7fa8dffff640 10 monclient(hunting): authenticate will time out at 1484.492234s 2024-03-29T10:57:51.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.257+0000 7fa8de7fc640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa8d8038dc0 0x7fa8d80cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51837/0 (socket says 172.21.15.5:51837) 2024-03-29T10:57:51.238 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.257+0000 7fa8de7fc640 1 -- 172.21.15.5:0/44257761 learned_addr learned my addr 172.21.15.5:0/44257761 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:51.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555997946 0 0) 0x7fa8d80c54c0 con 0x7fa8d8038dc0 2024-03-29T10:57:51.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:51.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:51.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:51.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient(hunting): my global_id is 4557 2024-03-29T10:57:51.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c8003520 con 0x7fa8d8038dc0 2024-03-29T10:57:51.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902718223 0 0) 0x7fa8d8039840 con 0x7fa8d8030660 2024-03-29T10:57:51.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:51.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:51.241 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient(hunting): my global_id is 4664 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d80c54c0 con 0x7fa8d8030660 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2536030708 0 0) 0x7fa8c8003520 con 0x7fa8d8038dc0 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 >> v1:172.21.15.5:6790/0 conn(0x7fa8d8030660 legacy=0x7fa8d80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 >> v1:172.21.15.177:6789/0 conn(0x7fa8d80389e0 legacy=0x7fa8d8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 1 monclient: found mon.noname-a 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8035690 con 0x7fa8d8038dc0 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:51.242 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa8e004f730 con 0x7fa8d8038dc0 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.258+0000 7fa8ddffb640 10 monclient: dump: 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.259+0000 7fa8dffff640 5 monclient: authenticate success, global_id 4557 2024-03-29T10:57:51.243 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.259+0000 7fa8dffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:51.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.262+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8e00008d0 con 0x7fa8d8038dc0 2024-03-29T10:57:51.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.262+0000 7fa8ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:51.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.262+0000 7fa8dffff640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:51.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.262+0000 7fa8dffff640 4 set_mon_vals no callback set 2024-03-29T10:57:51.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.263+0000 7fa8dffff640 10 monclient: shutdown 2024-03-29T10:57:51.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.263+0000 7fa8dffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:51.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.263+0000 7fa8dffff640 1 -- 172.21.15.5:0/44257761 >> v1:172.21.15.5:6789/0 conn(0x7fa8d8038dc0 legacy=0x7fa8d80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.263+0000 7fa8ddffb640 1 -- 172.21.15.5:0/44257761 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa8e00758b0 con 0x7fa8d8038dc0 2024-03-29T10:57:51.245 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.264+0000 7fa8ddffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:57:51.246 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.265+0000 7fa8dffff640 1 -- 172.21.15.5:0/44257761 shutdown_connections 2024-03-29T10:57:51.246 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.265+0000 7fa8dffff640 1 -- 172.21.15.5:0/44257761 >> 172.21.15.5:0/44257761 conn(0x7fa8d8005c00 msgr2=0x7fa8d8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:51.248 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.267+0000 7fa8dffff640 1 -- 172.21.15.5:0/44257761 shutdown_connections 2024-03-29T10:57:51.249 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.269+0000 7fa8dffff640 1 -- 172.21.15.5:0/44257761 wait complete. 2024-03-29T10:57:51.250 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.270+0000 7fa8dffff640 10 monclient: build_initial_monmap 2024-03-29T10:57:51.250 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.270+0000 7fa8dffff640 10 monclient: monmap: 2024-03-29T10:57:51.250 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:51.250 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:51.250 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:57:51.250 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:57:51.251 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:51.251 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:51.251 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:51.251 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:57:51.251 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:57:51.251 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:51.251 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.270+0000 7fa8dffff640 1 librados: starting msgr at 2024-03-29T10:57:51.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.271+0000 7fa8dffff640 1 librados: starting objecter 2024-03-29T10:57:51.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.271+0000 7fa8dffff640 1 Processor -- start 2024-03-29T10:57:51.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.271+0000 7fa8dffff640 1 -- start start 2024-03-29T10:57:51.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.271+0000 7fa8dffff640 1 librados: setting wanted keys 2024-03-29T10:57:51.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.271+0000 7fa8dffff640 1 librados: calling monclient init 2024-03-29T10:57:51.252 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.271+0000 7fa8dffff640 10 monclient: init 2024-03-29T10:57:51.253 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.272+0000 7fa8dffff640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.272+0000 7fa8dffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.272+0000 7fa8dffff640 10 monclient(hunting): picked mon.noname-c con 0x7fa8d8030660 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.272+0000 7fa8dffff640 10 monclient(hunting): picked mon.noname-a con 0x7fa8d80389e0 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.272+0000 7fa8dffff640 10 monclient(hunting): picked mon.noname-b con 0x7fa8d802fcf0 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.272+0000 7fa8dffff640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d80301a0 con 0x7fa8d80389e0 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.272+0000 7fa8dffff640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d80c65a0 con 0x7fa8d802fcf0 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.272+0000 7fa8dffff640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d80c7da0 con 0x7fa8d8030660 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.272+0000 7fa8dffff640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.272+0000 7fa8dffff640 10 monclient(hunting): authenticate will time out at 1484.514061s 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.276+0000 7fa8de7fc640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa8d80389e0 0x7fa8d80c5e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:52399/0 (socket says 172.21.15.5:52399) 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.276+0000 7fa8de7fc640 1 -- 172.21.15.5:0/2585532852 learned_addr learned my addr 172.21.15.5:0/2585532852 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:51.258 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.276+0000 7fa8c77fe640 10 client.?.objecter ms_handle_connect 0x7fa8d80389e0 2024-03-29T10:57:51.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.276+0000 7fa8c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:51.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.276+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246783580 0 0) 0x7fa8d80301a0 con 0x7fa8d80389e0 2024-03-29T10:57:51.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.276+0000 7fa8c77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:51.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.276+0000 7fa8c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:51.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.276+0000 7fa8c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:51.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.276+0000 7fa8c77fe640 10 monclient(hunting): my global_id is 4560 2024-03-29T10:57:51.259 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.276+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8cc004eb0 con 0x7fa8d80389e0 2024-03-29T10:57:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 client.?.objecter ms_handle_connect 0x7fa8d8030660 2024-03-29T10:57:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19854337 0 0) 0x7fa8d80c7da0 con 0x7fa8d8030660 2024-03-29T10:57:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 monclient(hunting): my global_id is 4667 2024-03-29T10:57:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d80301a0 con 0x7fa8d8030660 2024-03-29T10:57:51.266 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 client.?.objecter ms_handle_connect 0x7fa8d802fcf0 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2381766351 0 0) 0x7fa8cc004eb0 con 0x7fa8d80389e0 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8d80c7da0 con 0x7fa8d80389e0 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa8e004f400 con 0x7fa8d80389e0 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 monclient(hunting): dump: 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:51.267 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.283+0000 7fa8c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4014407343 0 0) 0x7fa8d80301a0 con 0x7fa8d8030660 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8cc004eb0 con 0x7fa8d8030660 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa8d4001510 con 0x7fa8d8030660 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient(hunting): dump: 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:51.268 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2025847663 0 0) 0x7fa8d80c7da0 con 0x7fa8d80389e0 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 >> v1:172.21.15.5:6790/0 conn(0x7fa8d8030660 legacy=0x7fa8d80c82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 >> v1:172.21.15.177:6789/0 conn(0x7fa8d802fcf0 legacy=0x7fa8d8159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 1 monclient: found mon.a 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d815a3d0 con 0x7fa8d80389e0 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:51.269 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8e004f730 con 0x7fa8d80389e0 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fa8e0067d90 con 0x7fa8d80389e0 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.284+0000 7fa8c77fe640 10 monclient: dump: 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:51.270 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:51.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.285+0000 7fa8dffff640 5 monclient: authenticate success, global_id 4560 2024-03-29T10:57:51.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.285+0000 7fa8dffff640 10 monclient: _renew_subs 2024-03-29T10:57:51.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.285+0000 7fa8dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:51.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.285+0000 7fa8dffff640 1 -- 172.21.15.5:0/2585532852 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d815a620 con 0x7fa8d80389e0 2024-03-29T10:57:51.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.285+0000 7fa8dffff640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:51.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.285+0000 7fa8dffff640 10 monclient: _renew_subs 2024-03-29T10:57:51.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.285+0000 7fa8dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:51.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.285+0000 7fa8dffff640 1 -- 172.21.15.5:0/2585532852 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d81601b0 con 0x7fa8d80389e0 2024-03-29T10:57:51.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.285+0000 7fa8dffff640 1 librados: init done 2024-03-29T10:57:51.271 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.285+0000 7fa8dd7fa640 4 set_mon_vals no callback set 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.288+0000 7fa8dffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.288+0000 7fa8dffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.288+0000 7fa8dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.288+0000 7fa8dffff640 1 -- 172.21.15.5:0/2585532852 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c0005080 con 0x7fa8d80389e0 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.289+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fa8e0067ef0 con 0x7fa8d80389e0 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.289+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3869902189 0 0) 0x7fa8e00ff3b0 con 0x7fa8d80389e0 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.289+0000 7fa8c77fe640 10 client.4560.objecter ms_dispatch 0x7fa8e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.289+0000 7fa8c77fe640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.289+0000 7fa8c77fe640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.289+0000 7fa8c77fe640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T10:57:51.273 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.289+0000 7fa8deffd640 10 monclient: get_auth_request con 0x7fa8cc06dee0 auth_method 0 2024-03-29T10:57:51.274 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.290+0000 7fa8c77fe640 10 client.4560.objecter ms_handle_connect 0x7fa8cc06dee0 2024-03-29T10:57:51.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.299+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8e00cac90 con 0x7fa8d80389e0 2024-03-29T10:57:51.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.299+0000 7fa8c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:51.280 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.299+0000 7fa8c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:51.615 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.631+0000 7fa8dffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:51.615 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.631+0000 7fa8dffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:51.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.631+0000 7fa8dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:57:51.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.631+0000 7fa8dffff640 1 -- 172.21.15.5:0/2585532852 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8c0005330 con 0x7fa8d80389e0 2024-03-29T10:57:51.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.632+0000 7fa8c77fe640 1 -- 172.21.15.5:0/2585532852 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 708563040) 0x7fa8e00ccbc0 con 0x7fa8d80389e0 2024-03-29T10:57:51.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.632+0000 7fa8c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:51.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.632+0000 7fa8c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:51.616 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:57:51.616 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:47:53.755595+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:53.755595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:51.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.635+0000 7fa8dffff640 10 librados: watch_flush enter 2024-03-29T10:57:51.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.635+0000 7fa8dffff640 10 librados: watch_flush exit 2024-03-29T10:57:51.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.635+0000 7fa8dffff640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T10:57:51.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.635+0000 7fa8dffff640 10 client.4560.objecter successfully canceled tick 2024-03-29T10:57:51.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.635+0000 7fa8dffff640 1 -- 172.21.15.5:0/2585532852 >> v1:172.21.15.5:6816/2847500151 conn(0x7fa8cc06dee0 legacy=0x7fa8cc0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.635+0000 7fa8dffff640 10 monclient: shutdown 2024-03-29T10:57:51.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.635+0000 7fa8dffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:51.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.635+0000 7fa8dffff640 1 -- 172.21.15.5:0/2585532852 >> v1:172.21.15.5:6789/0 conn(0x7fa8d80389e0 legacy=0x7fa8d80c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.636+0000 7fa8dffff640 1 -- 172.21.15.5:0/2585532852 shutdown_connections 2024-03-29T10:57:51.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.636+0000 7fa8dffff640 1 -- 172.21.15.5:0/2585532852 >> 172.21.15.5:0/2585532852 conn(0x7fa8d8005c00 msgr2=0x7fa8d80c4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:51.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.636+0000 7fa8dffff640 1 -- 172.21.15.5:0/2585532852 shutdown_connections 2024-03-29T10:57:51.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.636+0000 7fa8dffff640 1 -- 172.21.15.5:0/2585532852 wait complete. 2024-03-29T10:57:51.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:51.636+0000 7fa8dffff640 1 librados: shutdown 2024-03-29T10:57:51.631 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:58.438 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:58.586 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.605+0000 7f434d84c640 10 monclient: get_monmap_and_config 2024-03-29T10:57:58.586 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.605+0000 7f434d84c640 10 monclient: build_initial_monmap 2024-03-29T10:57:58.586 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.605+0000 7f434d84c640 10 monclient: monmap: 2024-03-29T10:57:58.586 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:58.587 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:58.587 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:57:58.606941+0000 2024-03-29T10:57:58.587 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:57:58.606941+0000 2024-03-29T10:57:58.587 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:58.587 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:58.587 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:58.587 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:57:58.587 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:57:58.587 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:58.589 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.609+0000 7f434d84c640 1 Processor -- start 2024-03-29T10:57:58.589 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.609+0000 7f434d84c640 1 -- start start 2024-03-29T10:57:58.589 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.609+0000 7f434d84c640 10 monclient: init 2024-03-29T10:57:58.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.610+0000 7f434d84c640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:58.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.610+0000 7f434d84c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:58.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.610+0000 7f434d84c640 10 monclient(hunting): picked mon.noname-a con 0x7f43480722c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:58.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.610+0000 7f434d84c640 10 monclient(hunting): picked mon.noname-c con 0x7f4348141e10 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:58.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.611+0000 7f434d84c640 10 monclient(hunting): picked mon.noname-b con 0x7f434814b690 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:58.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.611+0000 7f434d84c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43480aacb0 con 0x7f43480722c0 2024-03-29T10:57:58.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.611+0000 7f434d84c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43480aaf90 con 0x7f4348141e10 2024-03-29T10:57:58.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.611+0000 7f434d84c640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43480b0100 con 0x7f434814b690 2024-03-29T10:57:58.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.611+0000 7f434d84c640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:58.591 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.611+0000 7f434d84c640 10 monclient(hunting): authenticate will time out at 1491.852163s 2024-03-29T10:57:58.592 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.611+0000 7f4347fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f4348141e10 0x7f4348142270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:57004/0 (socket says 172.21.15.5:57004) 2024-03-29T10:57:58.592 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.611+0000 7f4347fff640 1 -- 172.21.15.5:0/329665734 learned_addr learned my addr 172.21.15.5:0/329665734 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:58.594 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.614+0000 7f43477fe640 1 -- 172.21.15.5:0/329665734 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359507156 0 0) 0x7f43480aaf90 con 0x7f4348141e10 2024-03-29T10:57:58.594 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.614+0000 7f43477fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:58.594 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.614+0000 7f43477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:58.594 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.614+0000 7f43477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:58.594 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.614+0000 7f43477fe640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:57:58.595 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.614+0000 7f43477fe640 1 -- 172.21.15.5:0/329665734 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4330003520 con 0x7f4348141e10 2024-03-29T10:57:58.595 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.614+0000 7f43477fe640 1 -- 172.21.15.5:0/329665734 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3183025998 0 0) 0x7f4330003520 con 0x7f4348141e10 2024-03-29T10:57:58.595 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.614+0000 7f43477fe640 1 -- 172.21.15.5:0/329665734 >> v1:172.21.15.5:6790/0 conn(0x7f434814b690 legacy=0x7f434814da80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 1 -- 172.21.15.5:0/329665734 >> v1:172.21.15.5:6789/0 conn(0x7f43480722c0 legacy=0x7f434813eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 1 monclient: found mon.noname-c 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 1 -- 172.21.15.5:0/329665734 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434813f8c0 con 0x7f4348141e10 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 1 -- 172.21.15.5:0/329665734 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4338001690 con 0x7f4348141e10 2024-03-29T10:57:58.597 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:58.598 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:58.598 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f43477fe640 10 monclient: dump: 2024-03-29T10:57:58.598 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:58.598 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:58.598 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:58.598 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:58.598 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:58.598 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:58.599 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:58.599 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:58.599 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:58.599 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:58.599 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f434d84c640 5 monclient: authenticate success, global_id 4492 2024-03-29T10:57:58.599 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.616+0000 7f434d84c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:58.600 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.617+0000 7f43477fe640 1 -- 172.21.15.5:0/329665734 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4338001860 con 0x7f4348141e10 2024-03-29T10:57:58.600 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.617+0000 7f43477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:58.600 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.617+0000 7f43477fe640 1 -- 172.21.15.5:0/329665734 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4338001c70 con 0x7f4348141e10 2024-03-29T10:57:58.600 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.617+0000 7f43477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.617+0000 7f43477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.617+0000 7f43477fe640 10 monclient: dump: 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:58.601 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:58.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.618+0000 7f434d84c640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:58.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.618+0000 7f434d84c640 4 set_mon_vals no callback set 2024-03-29T10:57:58.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.618+0000 7f434d84c640 10 monclient: shutdown 2024-03-29T10:57:58.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.618+0000 7f434d84c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:58.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.618+0000 7f434d84c640 1 -- 172.21.15.5:0/329665734 >> v1:172.21.15.177:6789/0 conn(0x7f4348141e10 legacy=0x7f4348142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:58.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.620+0000 7f434d84c640 1 -- 172.21.15.5:0/329665734 shutdown_connections 2024-03-29T10:57:58.602 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.620+0000 7f434d84c640 1 -- 172.21.15.5:0/329665734 >> 172.21.15.5:0/329665734 conn(0x7f43480b3ae0 msgr2=0x7f43480b3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:58.603 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.621+0000 7f434d84c640 1 -- 172.21.15.5:0/329665734 shutdown_connections 2024-03-29T10:57:58.603 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.621+0000 7f434d84c640 1 -- 172.21.15.5:0/329665734 wait complete. 2024-03-29T10:57:58.605 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.622+0000 7f434d84c640 10 monclient: build_initial_monmap 2024-03-29T10:57:58.605 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.622+0000 7f434d84c640 10 monclient: monmap: 2024-03-29T10:57:58.605 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:57:58.605 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:58.605 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:57:58.605 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:57:58.605 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:58.605 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:58.605 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.622+0000 7f434d84c640 1 librados: starting msgr at 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.622+0000 7f434d84c640 1 librados: starting objecter 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.622+0000 7f434d84c640 1 Processor -- start 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.622+0000 7f434d84c640 1 -- start start 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.622+0000 7f434d84c640 1 librados: setting wanted keys 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.622+0000 7f434d84c640 1 librados: calling monclient init 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.622+0000 7f434d84c640 10 monclient: init 2024-03-29T10:57:58.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.623+0000 7f434d84c640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:58.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.623+0000 7f434d84c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:58.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.623+0000 7f434d84c640 10 monclient(hunting): picked mon.noname-c con 0x7f43480722c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:57:58.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.623+0000 7f434d84c640 10 monclient(hunting): picked mon.noname-a con 0x7f4348141e10 addr v1:172.21.15.5:6789/0 2024-03-29T10:57:58.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.624+0000 7f434d84c640 10 monclient(hunting): picked mon.noname-b con 0x7f434814b690 addr v1:172.21.15.177:6789/0 2024-03-29T10:57:58.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.624+0000 7f434d84c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434814b4d0 con 0x7f4348141e10 2024-03-29T10:57:58.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.624+0000 7f434d84c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4348071120 con 0x7f434814b690 2024-03-29T10:57:58.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.624+0000 7f434d84c640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4348071400 con 0x7f43480722c0 2024-03-29T10:57:58.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.624+0000 7f434d84c640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:58.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.624+0000 7f434d84c640 10 monclient(hunting): authenticate will time out at 1491.865483s 2024-03-29T10:57:58.608 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.624+0000 7f434d04b640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f434814b690 0x7f434806e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:58132/0 (socket says 172.21.15.5:58132) 2024-03-29T10:57:58.608 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.624+0000 7f434d04b640 1 -- 172.21.15.5:0/3387134939 learned_addr learned my addr 172.21.15.5:0/3387134939 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:57:58.608 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.627+0000 7f43457fa640 10 client.?.objecter ms_handle_connect 0x7f434814b690 2024-03-29T10:57:58.609 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.627+0000 7f43457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:58.609 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.627+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843429983 0 0) 0x7f4348071120 con 0x7f434814b690 2024-03-29T10:57:58.609 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.627+0000 7f43457fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:58.609 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.627+0000 7f43457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:58.609 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.627+0000 7f43457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:58.609 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.627+0000 7f43457fa640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:57:58.609 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.627+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4334004eb0 con 0x7f434814b690 2024-03-29T10:57:58.609 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.629+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1904740031 0 0) 0x7f4334004eb0 con 0x7f434814b690 2024-03-29T10:57:58.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.629+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4348071120 con 0x7f434814b690 2024-03-29T10:57:58.610 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.629+0000 7f43457fa640 10 client.?.objecter ms_handle_connect 0x7f4348141e10 2024-03-29T10:57:58.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:58.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 client.?.objecter ms_handle_connect 0x7f43480722c0 2024-03-29T10:57:58.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:58.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4340004280 con 0x7f434814b690 2024-03-29T10:57:58.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:58.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:58.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient(hunting): dump: 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611836920 0 0) 0x7f434814b4d0 con 0x7f4348141e10 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:58.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4334004eb0 con 0x7f4348141e10 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3021418730 0 0) 0x7f4348071120 con 0x7f434814b690 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 >> v1:172.21.15.5:6790/0 conn(0x7f43480722c0 legacy=0x7f4348071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 >> v1:172.21.15.5:6789/0 conn(0x7f4348141e10 legacy=0x7f434806dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient: _finish_hunting 0 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 1 monclient: found mon.b 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434806ea60 con 0x7f434814b690 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient: _finish_auth 0 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 10 monclient: _check_auth_tickets 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f43457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f434d84c640 5 monclient: authenticate success, global_id 4495 2024-03-29T10:57:58.618 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f434d84c640 10 monclient: _renew_subs 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f434d84c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f434d84c640 1 -- 172.21.15.5:0/3387134939 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434814b180 con 0x7f434814b690 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f434d84c640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f434d84c640 10 monclient: _renew_subs 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f434d84c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f434d84c640 1 -- 172.21.15.5:0/3387134939 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4348139c80 con 0x7f434814b690 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.632+0000 7f434d84c640 1 librados: init done 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.634+0000 7f434d84c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.634+0000 7f434d84c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.634+0000 7f434d84c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.634+0000 7f434d84c640 1 -- 172.21.15.5:0/3387134939 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434806ec50 con 0x7f434814b690 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.634+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43400043e0 con 0x7f434814b690 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.635+0000 7f43457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.635+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f43400013f0 con 0x7f434814b690 2024-03-29T10:57:58.619 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.635+0000 7f4346ffd640 4 set_mon_vals no callback set 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.636+0000 7f43457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.636+0000 7f43457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.636+0000 7f43457fa640 10 monclient: dump: 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:57:58.621 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:57:58.622 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:57:58.622 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:57:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.637+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f4340001550 con 0x7f434814b690 2024-03-29T10:57:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.638+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1216745598 0 0) 0x7f4340084a70 con 0x7f434814b690 2024-03-29T10:57:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.638+0000 7f43457fa640 10 client.4495.objecter ms_dispatch 0x7f4348000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:57:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.638+0000 7f43457fa640 3 client.4495.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:57:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.638+0000 7f43457fa640 3 client.4495.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:57:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.638+0000 7f43457fa640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T10:57:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.639+0000 7f434c84a640 10 monclient: get_auth_request con 0x7f433406dcb0 auth_method 0 2024-03-29T10:57:58.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.639+0000 7f43457fa640 10 client.4495.objecter ms_handle_connect 0x7f433406dcb0 2024-03-29T10:57:58.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.643+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4340050350 con 0x7f434814b690 2024-03-29T10:57:58.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.643+0000 7f43457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:58.623 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.643+0000 7f43457fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:58.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.977+0000 7f434d84c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:58.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.977+0000 7f434d84c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:58.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.977+0000 7f434d84c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:57:58.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.977+0000 7f434d84c640 1 -- 172.21.15.5:0/3387134939 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f434806ebf0 con 0x7f434814b690 2024-03-29T10:57:58.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.978+0000 7f43457fa640 1 -- 172.21.15.5:0/3387134939 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 16110640) 0x7f4340052280 con 0x7f434814b690 2024-03-29T10:57:58.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.978+0000 7f43457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:58.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.978+0000 7f43457fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:58.960 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:57:58.960 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:57:56.452759+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:56.452759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:58.963 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.982+0000 7f434d84c640 10 librados: watch_flush enter 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.982+0000 7f434d84c640 10 librados: watch_flush exit 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.982+0000 7f434d84c640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.982+0000 7f434d84c640 10 client.4495.objecter successfully canceled tick 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.983+0000 7f434d84c640 1 -- 172.21.15.5:0/3387134939 >> v1:172.21.15.5:6816/2847500151 conn(0x7f433406dcb0 legacy=0x7f4334070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.983+0000 7f434d84c640 10 monclient: shutdown 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.983+0000 7f434d84c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.983+0000 7f434d84c640 1 -- 172.21.15.5:0/3387134939 >> v1:172.21.15.177:6789/0 conn(0x7f434814b690 legacy=0x7f434806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.983+0000 7f434d84c640 1 -- 172.21.15.5:0/3387134939 shutdown_connections 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.983+0000 7f434d84c640 1 -- 172.21.15.5:0/3387134939 >> 172.21.15.5:0/3387134939 conn(0x7f43480b3ae0 msgr2=0x7f4348069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.983+0000 7f434d84c640 1 -- 172.21.15.5:0/3387134939 shutdown_connections 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.983+0000 7f434d84c640 1 -- 172.21.15.5:0/3387134939 wait complete. 2024-03-29T10:57:58.964 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:57:58.983+0000 7f434d84c640 1 librados: shutdown 2024-03-29T10:57:58.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:05.788 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:05.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.961+0000 7efd3f222640 10 monclient: get_monmap_and_config 2024-03-29T10:58:05.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.961+0000 7efd3f222640 10 monclient: build_initial_monmap 2024-03-29T10:58:05.943 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.961+0000 7efd3f222640 10 monclient: monmap: 2024-03-29T10:58:05.943 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:05.943 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:05.943 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:58:05.962916+0000 2024-03-29T10:58:05.943 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:58:05.962916+0000 2024-03-29T10:58:05.943 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:05.944 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:05.944 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:05.944 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:58:05.944 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:58:05.944 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:05.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.962+0000 7efd3f222640 1 Processor -- start 2024-03-29T10:58:05.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.962+0000 7efd3f222640 1 -- start start 2024-03-29T10:58:05.944 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.962+0000 7efd3f222640 10 monclient: init 2024-03-29T10:58:05.945 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.964+0000 7efd3f222640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:05.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.964+0000 7efd3f222640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:05.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.964+0000 7efd3f222640 10 monclient(hunting): picked mon.noname-a con 0x7efd380b2800 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:05.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.964+0000 7efd3f222640 10 monclient(hunting): picked mon.noname-c con 0x7efd380a9c00 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:05.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.965+0000 7efd3f222640 10 monclient(hunting): picked mon.noname-b con 0x7efd380b38c0 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:05.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.965+0000 7efd3f222640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd38141eb0 con 0x7efd380b2800 2024-03-29T10:58:05.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.965+0000 7efd3f222640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd380b22d0 con 0x7efd380a9c00 2024-03-29T10:58:05.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.965+0000 7efd3f222640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd38142d30 con 0x7efd380b38c0 2024-03-29T10:58:05.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.965+0000 7efd3f222640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:05.946 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.965+0000 7efd3f222640 10 monclient(hunting): authenticate will time out at 1499.206752s 2024-03-29T10:58:05.949 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.967+0000 7efd3ea21640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7efd380b38c0 0x7efd3814d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:59596/0 (socket says 172.21.15.5:59596) 2024-03-29T10:58:05.949 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.967+0000 7efd3ea21640 1 -- 172.21.15.5:0/2554875508 learned_addr learned my addr 172.21.15.5:0/2554875508 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.968+0000 7efd3d21e640 1 -- 172.21.15.5:0/2554875508 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244863492 0 0) 0x7efd380b22d0 con 0x7efd380a9c00 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.968+0000 7efd3d21e640 10 monclient(hunting): _negotiate 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.968+0000 7efd3d21e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.968+0000 7efd3d21e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.968+0000 7efd3d21e640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.968+0000 7efd3d21e640 1 -- 172.21.15.5:0/2554875508 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd24003520 con 0x7efd380a9c00 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.968+0000 7efd3d21e640 1 -- 172.21.15.5:0/2554875508 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4166530400 0 0) 0x7efd24003520 con 0x7efd380a9c00 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.968+0000 7efd3d21e640 1 -- 172.21.15.5:0/2554875508 >> v1:172.21.15.5:6790/0 conn(0x7efd380b38c0 legacy=0x7efd3814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.968+0000 7efd3d21e640 1 -- 172.21.15.5:0/2554875508 >> v1:172.21.15.5:6789/0 conn(0x7efd380b2800 legacy=0x7efd38142620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: _finish_hunting 0 2024-03-29T10:58:05.950 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 1 monclient: found mon.noname-c 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.177:6789/0 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 1 -- 172.21.15.5:0/2554875508 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd380aaf90 con 0x7efd380a9c00 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: _finish_auth 0 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: _check_auth_tickets 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 1 -- 172.21.15.5:0/2554875508 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efd2c001690 con 0x7efd380a9c00 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: dump: 2024-03-29T10:58:05.951 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3f222640 5 monclient: authenticate success, global_id 4498 2024-03-29T10:58:05.952 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3f222640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 1 -- 172.21.15.5:0/2554875508 <== mon.1 v1:172.21.15.177:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd2c0017f0 con 0x7efd380a9c00 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 1 -- 172.21.15.5:0/2554875508 <== mon.1 v1:172.21.15.177:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efd2c001c40 con 0x7efd380a9c00 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.969+0000 7efd3d21e640 10 monclient: dump: 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:05.955 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.971+0000 7efd3f222640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.971+0000 7efd3f222640 4 set_mon_vals no callback set 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.972+0000 7efd3f222640 10 monclient: shutdown 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.972+0000 7efd3f222640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.972+0000 7efd3f222640 1 -- 172.21.15.5:0/2554875508 >> v1:172.21.15.177:6789/0 conn(0x7efd380a9c00 legacy=0x7efd3813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.973+0000 7efd3f222640 1 -- 172.21.15.5:0/2554875508 shutdown_connections 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.973+0000 7efd3f222640 1 -- 172.21.15.5:0/2554875508 >> 172.21.15.5:0/2554875508 conn(0x7efd38066a00 msgr2=0x7efd38066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.974+0000 7efd3f222640 1 -- 172.21.15.5:0/2554875508 shutdown_connections 2024-03-29T10:58:05.956 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.974+0000 7efd3f222640 1 -- 172.21.15.5:0/2554875508 wait complete. 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.975+0000 7efd3f222640 10 monclient: build_initial_monmap 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.975+0000 7efd3f222640 10 monclient: monmap: 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:58:05.958 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:58:05.959 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:05.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.976+0000 7efd3f222640 1 librados: starting msgr at 2024-03-29T10:58:05.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.976+0000 7efd3f222640 1 librados: starting objecter 2024-03-29T10:58:05.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.976+0000 7efd3f222640 1 Processor -- start 2024-03-29T10:58:05.959 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.976+0000 7efd3f222640 1 -- start start 2024-03-29T10:58:05.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.976+0000 7efd3f222640 1 librados: setting wanted keys 2024-03-29T10:58:05.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.976+0000 7efd3f222640 1 librados: calling monclient init 2024-03-29T10:58:05.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.976+0000 7efd3f222640 10 monclient: init 2024-03-29T10:58:05.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.977+0000 7efd3f222640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:05.960 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.977+0000 7efd3f222640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.980+0000 7efd3f222640 10 monclient(hunting): picked mon.noname-b con 0x7efd380a9c00 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.980+0000 7efd3f222640 10 monclient(hunting): picked mon.noname-c con 0x7efd380b2800 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.980+0000 7efd3f222640 10 monclient(hunting): picked mon.noname-a con 0x7efd380b38c0 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.980+0000 7efd3f222640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd380af6d0 con 0x7efd380b38c0 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.980+0000 7efd3f222640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd380af900 con 0x7efd380a9c00 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.981+0000 7efd3f222640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd380afbe0 con 0x7efd380b2800 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.981+0000 7efd3f222640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.981+0000 7efd3f222640 10 monclient(hunting): authenticate will time out at 1499.222102s 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.981+0000 7efd3da1f640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7efd380b2800 0x7efd380b07c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:60006/0 (socket says 172.21.15.5:60006) 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.981+0000 7efd3da1f640 1 -- 172.21.15.5:0/2889861483 learned_addr learned my addr 172.21.15.5:0/2889861483 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:05.962 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.981+0000 7efd2affd640 10 client.?.objecter ms_handle_connect 0x7efd380b2800 2024-03-29T10:58:05.963 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.981+0000 7efd2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:05.970 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907635484 0 0) 0x7efd380afbe0 con 0x7efd380b2800 2024-03-29T10:58:05.970 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:05.970 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:05.970 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:05.970 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 10 monclient(hunting): my global_id is 4673 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd1c004eb0 con 0x7efd380b2800 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 10 client.?.objecter ms_handle_connect 0x7efd380a9c00 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141665913 0 0) 0x7efd1c004eb0 con 0x7efd380b2800 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd380afbe0 con 0x7efd380b2800 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efd2c001b80 con 0x7efd380b2800 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 10 monclient(hunting): dump: 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.971 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.982+0000 7efd2affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.983+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1078836897 0 0) 0x7efd380afbe0 con 0x7efd380b2800 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.983+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 >> v1:172.21.15.177:6789/0 conn(0x7efd380a9c00 legacy=0x7efd380ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.983+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 >> v1:172.21.15.5:6789/0 conn(0x7efd380b38c0 legacy=0x7efd380b0ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.984+0000 7efd2affd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.984+0000 7efd2affd640 1 monclient: found mon.c 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.984+0000 7efd2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.984+0000 7efd2affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:05.972 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.984+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd380ae930 con 0x7efd380b2800 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.984+0000 7efd2affd640 10 monclient: _finish_auth 0 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.984+0000 7efd2affd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.984+0000 7efd2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd3f222640 5 monclient: authenticate success, global_id 4673 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd3f222640 10 monclient: _renew_subs 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd2c003940 con 0x7efd380b2800 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd3f222640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd3f222640 1 -- 172.21.15.5:0/2889861483 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd380aeb80 con 0x7efd380b2800 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efd2c003d50 con 0x7efd380b2800 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd2affd640 10 monclient: dump: 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:05.973 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd2bfff640 4 set_mon_vals no callback set 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd3f222640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd3f222640 10 monclient: _renew_subs 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd3f222640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd3f222640 1 -- 172.21.15.5:0/2889861483 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd381e2af0 con 0x7efd380b2800 2024-03-29T10:58:05.974 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.985+0000 7efd3f222640 1 librados: init done 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.987+0000 7efd28ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.987+0000 7efd28ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.987+0000 7efd28ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.987+0000 7efd28ff9640 1 -- 172.21.15.5:0/2889861483 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd380b04c0 con 0x7efd380b2800 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.988+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7efd2c003eb0 con 0x7efd380b2800 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.989+0000 7efd3e220640 10 monclient: get_auth_request con 0x7efd1c06dd00 auth_method 0 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.989+0000 7efd2affd640 10 client.4673.objecter ms_handle_connect 0x7efd1c06dd00 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.989+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1216745598 0 0) 0x7efd2c054a60 con 0x7efd380b2800 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.989+0000 7efd2affd640 10 client.4673.objecter ms_dispatch 0x7efd38000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.989+0000 7efd2affd640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.989+0000 7efd2affd640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:05.975 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.989+0000 7efd2affd640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T10:58:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.996+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd2c050570 con 0x7efd380b2800 2024-03-29T10:58:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.996+0000 7efd2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:05.976 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:05.996+0000 7efd2affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:06.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.327+0000 7efd28ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:06.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.327+0000 7efd28ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:06.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.327+0000 7efd28ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:06.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.327+0000 7efd28ff9640 1 -- 172.21.15.5:0/2889861483 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd3804f730 con 0x7efd380b2800 2024-03-29T10:58:06.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.330+0000 7efd2affd640 1 -- 172.21.15.5:0/2889861483 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 16110640) 0x7efd2c0524a0 con 0x7efd380b2800 2024-03-29T10:58:06.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.330+0000 7efd2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:06.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.330+0000 7efd2affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:06.311 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:58:06.311 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:57:56.452759+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:56.452759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:06.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.333+0000 7efd28ff9640 10 librados: watch_flush enter 2024-03-29T10:58:06.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.333+0000 7efd28ff9640 10 librados: watch_flush exit 2024-03-29T10:58:06.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.333+0000 7efd28ff9640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T10:58:06.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.334+0000 7efd28ff9640 10 client.4673.objecter successfully canceled tick 2024-03-29T10:58:06.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.334+0000 7efd28ff9640 1 -- 172.21.15.5:0/2889861483 >> v1:172.21.15.5:6816/2847500151 conn(0x7efd1c06dd00 legacy=0x7efd1c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:06.315 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.334+0000 7efd28ff9640 10 monclient: shutdown 2024-03-29T10:58:06.315 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.334+0000 7efd28ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:06.315 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.334+0000 7efd28ff9640 1 -- 172.21.15.5:0/2889861483 >> v1:172.21.15.5:6790/0 conn(0x7efd380b2800 legacy=0x7efd380b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:06.315 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.334+0000 7efd28ff9640 1 -- 172.21.15.5:0/2889861483 shutdown_connections 2024-03-29T10:58:06.315 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.334+0000 7efd28ff9640 1 -- 172.21.15.5:0/2889861483 >> 172.21.15.5:0/2889861483 conn(0x7efd38066a00 msgr2=0x7efd380b82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:06.315 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.334+0000 7efd28ff9640 1 -- 172.21.15.5:0/2889861483 shutdown_connections 2024-03-29T10:58:06.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.334+0000 7efd28ff9640 1 -- 172.21.15.5:0/2889861483 wait complete. 2024-03-29T10:58:06.316 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:06.334+0000 7efd28ff9640 1 librados: shutdown 2024-03-29T10:58:06.332 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:13.139 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:13.286 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.306+0000 7efe4f513640 10 monclient: get_monmap_and_config 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.306+0000 7efe4f513640 10 monclient: build_initial_monmap 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.306+0000 7efe4f513640 10 monclient: monmap: 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:58:13.307265+0000 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:58:13.307265+0000 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:58:13.287 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:13.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.307+0000 7efe4f513640 1 Processor -- start 2024-03-29T10:58:13.288 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.307+0000 7efe4f513640 1 -- start start 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.307+0000 7efe4f513640 10 monclient: init 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.308+0000 7efe4f513640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.308+0000 7efe4f513640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.308+0000 7efe4f513640 10 monclient(hunting): picked mon.noname-b con 0x7efe480b0770 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.308+0000 7efe4f513640 10 monclient(hunting): picked mon.noname-c con 0x7efe4814a700 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.308+0000 7efe4f513640 10 monclient(hunting): picked mon.noname-a con 0x7efe480ac5a0 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.308+0000 7efe4f513640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe480aba70 con 0x7efe480ac5a0 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.308+0000 7efe4f513640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe480aecb0 con 0x7efe4814a700 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.308+0000 7efe4f513640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe480af150 con 0x7efe480b0770 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.308+0000 7efe4f513640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:13.289 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.308+0000 7efe4f513640 10 monclient(hunting): authenticate will time out at 1506.549681s 2024-03-29T10:58:13.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.309+0000 7efe4d2af640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7efe480b0770 0x7efe48149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:46774/0 (socket says 172.21.15.5:46774) 2024-03-29T10:58:13.290 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.309+0000 7efe4d2af640 1 -- 172.21.15.5:0/705244166 learned_addr learned my addr 172.21.15.5:0/705244166 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:13.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481612473 0 0) 0x7efe480af150 con 0x7efe480b0770 2024-03-29T10:58:13.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:13.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:13.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:13.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 10 monclient(hunting): my global_id is 4676 2024-03-29T10:58:13.294 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe34003520 con 0x7efe480b0770 2024-03-29T10:58:13.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731244661 0 0) 0x7efe480aba70 con 0x7efe480ac5a0 2024-03-29T10:58:13.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:13.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:13.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:13.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 10 monclient(hunting): my global_id is 4569 2024-03-29T10:58:13.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.312+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe480af150 con 0x7efe480ac5a0 2024-03-29T10:58:13.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604447488 0 0) 0x7efe480af150 con 0x7efe480ac5a0 2024-03-29T10:58:13.295 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 >> v1:172.21.15.5:6790/0 conn(0x7efe480b0770 legacy=0x7efe48149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:13.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 >> v1:172.21.15.177:6789/0 conn(0x7efe4814a700 legacy=0x7efe480abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:13.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:13.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 1 monclient: found mon.noname-a 2024-03-29T10:58:13.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:13.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:58:13.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe480ac9d0 con 0x7efe480ac5a0 2024-03-29T10:58:13.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 10 monclient: _finish_auth 0 2024-03-29T10:58:13.296 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:13.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:13.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efe38001670 con 0x7efe480ac5a0 2024-03-29T10:58:13.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:13.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:13.297 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 10 monclient: dump: 2024-03-29T10:58:13.297 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:13.297 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:13.297 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:13.297 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:13.297 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe4f513640 5 monclient: authenticate success, global_id 4569 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe4f513640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe38001890 con 0x7efe480ac5a0 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.313+0000 7efe3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.314+0000 7efe3ffff640 1 -- 172.21.15.5:0/705244166 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efe38001c20 con 0x7efe480ac5a0 2024-03-29T10:58:13.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.314+0000 7efe3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.314+0000 7efe3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.314+0000 7efe3ffff640 10 monclient: dump: 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.314+0000 7efe4f513640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:13.299 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.314+0000 7efe4f513640 4 set_mon_vals no callback set 2024-03-29T10:58:13.302 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.315+0000 7efe4f513640 10 monclient: shutdown 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.315+0000 7efe4f513640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.315+0000 7efe4f513640 1 -- 172.21.15.5:0/705244166 >> v1:172.21.15.5:6789/0 conn(0x7efe480ac5a0 legacy=0x7efe48151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.315+0000 7efe4f513640 1 -- 172.21.15.5:0/705244166 shutdown_connections 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.315+0000 7efe4f513640 1 -- 172.21.15.5:0/705244166 >> 172.21.15.5:0/705244166 conn(0x7efe4806b660 msgr2=0x7efe4806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.316+0000 7efe4f513640 1 -- 172.21.15.5:0/705244166 shutdown_connections 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.318+0000 7efe4f513640 1 -- 172.21.15.5:0/705244166 wait complete. 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.319+0000 7efe4f513640 10 monclient: build_initial_monmap 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.319+0000 7efe4f513640 10 monclient: monmap: 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:13.303 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.319+0000 7efe4f513640 1 librados: starting msgr at 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.319+0000 7efe4f513640 1 librados: starting objecter 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.319+0000 7efe4f513640 1 Processor -- start 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.319+0000 7efe4f513640 1 -- start start 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.319+0000 7efe4f513640 1 librados: setting wanted keys 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.319+0000 7efe4f513640 1 librados: calling monclient init 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.319+0000 7efe4f513640 10 monclient: init 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.320+0000 7efe4f513640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.320+0000 7efe4f513640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.320+0000 7efe4f513640 10 monclient(hunting): picked mon.noname-a con 0x7efe480ac5a0 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.320+0000 7efe4f513640 10 monclient(hunting): picked mon.noname-b con 0x7efe480b0770 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:13.304 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.320+0000 7efe4f513640 10 monclient(hunting): picked mon.noname-c con 0x7efe4814a700 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.320+0000 7efe4f513640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe481285a0 con 0x7efe480ac5a0 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.320+0000 7efe4f513640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe48126270 con 0x7efe480b0770 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.320+0000 7efe4f513640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe48145ce0 con 0x7efe4814a700 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.320+0000 7efe4f513640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.320+0000 7efe4f513640 10 monclient(hunting): authenticate will time out at 1506.561640s 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.321+0000 7efe4dab0640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7efe4814a700 0x7efe48126ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:47572/0 (socket says 172.21.15.5:47572) 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.321+0000 7efe4dab0640 1 -- 172.21.15.5:0/3791139414 learned_addr learned my addr 172.21.15.5:0/3791139414 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.321+0000 7efe3dffb640 10 client.?.objecter ms_handle_connect 0x7efe4814a700 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.321+0000 7efe3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.322+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407586225 0 0) 0x7efe48145ce0 con 0x7efe4814a700 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.322+0000 7efe3dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.322+0000 7efe3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.322+0000 7efe3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.322+0000 7efe3dffb640 10 monclient(hunting): my global_id is 4679 2024-03-29T10:58:13.305 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.322+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe2c004eb0 con 0x7efe4814a700 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 10 client.?.objecter ms_handle_connect 0x7efe480ac5a0 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290194632 0 0) 0x7efe2c004eb0 con 0x7efe4814a700 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe48145ce0 con 0x7efe4814a700 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efe380016b0 con 0x7efe4814a700 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 10 monclient(hunting): dump: 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:13.307 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:13.308 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:13.308 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:13.308 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:13.308 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:13.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:13.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 290960871 0 0) 0x7efe48145ce0 con 0x7efe4814a700 2024-03-29T10:58:13.308 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.323+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 >> v1:172.21.15.177:6789/0 conn(0x7efe480b0770 legacy=0x7efe48126790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:13.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.327+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 >> v1:172.21.15.5:6789/0 conn(0x7efe480ac5a0 legacy=0x7efe48144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:13.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.327+0000 7efe3dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:13.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.327+0000 7efe3dffb640 1 monclient: found mon.c 2024-03-29T10:58:13.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.327+0000 7efe3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:13.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.327+0000 7efe3dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:13.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.327+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe48125ed0 con 0x7efe4814a700 2024-03-29T10:58:13.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.327+0000 7efe3dffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:13.310 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.327+0000 7efe3dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.327+0000 7efe3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe4f513640 5 monclient: authenticate success, global_id 4679 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe4f513640 10 monclient: _renew_subs 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe4f513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe4f513640 1 -- 172.21.15.5:0/3791139414 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe481df650 con 0x7efe4814a700 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe4f513640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe4f513640 10 monclient: _renew_subs 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe4f513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe380038a0 con 0x7efe4814a700 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe4f513640 1 -- 172.21.15.5:0/3791139414 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe481dedf0 con 0x7efe4814a700 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe4f513640 1 librados: init done 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7efe38003cb0 con 0x7efe4814a700 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:13.311 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe3dffb640 10 monclient: dump: 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.328+0000 7efe3effd640 4 set_mon_vals no callback set 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.329+0000 7efe4f513640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.329+0000 7efe4f513640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.329+0000 7efe4f513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:13.312 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.330+0000 7efe4f513640 1 -- 172.21.15.5:0/3791139414 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe28005080 con 0x7efe4814a700 2024-03-29T10:58:13.313 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.330+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7efe38003e10 con 0x7efe4814a700 2024-03-29T10:58:13.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.331+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1216745598 0 0) 0x7efe38084c20 con 0x7efe4814a700 2024-03-29T10:58:13.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.331+0000 7efe3dffb640 10 client.4679.objecter ms_dispatch 0x7efe48000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:13.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.331+0000 7efe3dffb640 3 client.4679.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:13.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.331+0000 7efe3dffb640 3 client.4679.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:13.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.331+0000 7efe3dffb640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-29T10:58:13.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.332+0000 7efe4d2af640 10 monclient: get_auth_request con 0x7efe2c06dbb0 auth_method 0 2024-03-29T10:58:13.314 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.332+0000 7efe3dffb640 10 client.4679.objecter ms_handle_connect 0x7efe2c06dbb0 2024-03-29T10:58:13.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.340+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe38050500 con 0x7efe4814a700 2024-03-29T10:58:13.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.340+0000 7efe3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:13.321 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.340+0000 7efe3dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:13.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.674+0000 7efe4f513640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:13.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.674+0000 7efe4f513640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:13.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.674+0000 7efe4f513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:13.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.674+0000 7efe4f513640 1 -- 172.21.15.5:0/3791139414 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe28005330 con 0x7efe4814a700 2024-03-29T10:58:13.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.675+0000 7efe3dffb640 1 -- 172.21.15.5:0/3791139414 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 16110640) 0x7efe38052430 con 0x7efe4814a700 2024-03-29T10:58:13.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.675+0000 7efe3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:13.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.675+0000 7efe3dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:13.656 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:58:13.656 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:57:56.452759+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:56.452759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:13.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.678+0000 7efe4f513640 10 librados: watch_flush enter 2024-03-29T10:58:13.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.679+0000 7efe4f513640 10 librados: watch_flush exit 2024-03-29T10:58:13.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.679+0000 7efe4f513640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-29T10:58:13.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.679+0000 7efe4f513640 10 client.4679.objecter successfully canceled tick 2024-03-29T10:58:13.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.679+0000 7efe4f513640 1 -- 172.21.15.5:0/3791139414 >> v1:172.21.15.5:6816/2847500151 conn(0x7efe2c06dbb0 legacy=0x7efe2c070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:13.659 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.679+0000 7efe4f513640 10 monclient: shutdown 2024-03-29T10:58:13.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.679+0000 7efe4f513640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:13.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.679+0000 7efe4f513640 1 -- 172.21.15.5:0/3791139414 >> v1:172.21.15.5:6790/0 conn(0x7efe4814a700 legacy=0x7efe48126ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:13.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.679+0000 7efe4f513640 1 -- 172.21.15.5:0/3791139414 shutdown_connections 2024-03-29T10:58:13.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.679+0000 7efe4f513640 1 -- 172.21.15.5:0/3791139414 >> 172.21.15.5:0/3791139414 conn(0x7efe4806b660 msgr2=0x7efe4806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:13.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.680+0000 7efe4f513640 1 -- 172.21.15.5:0/3791139414 shutdown_connections 2024-03-29T10:58:13.660 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.680+0000 7efe4f513640 1 -- 172.21.15.5:0/3791139414 wait complete. 2024-03-29T10:58:13.661 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:13.680+0000 7efe4f513640 1 librados: shutdown 2024-03-29T10:58:13.677 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:20.485 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.650+0000 7fcce4eae640 10 monclient: get_monmap_and_config 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.650+0000 7fcce4eae640 10 monclient: build_initial_monmap 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.650+0000 7fcce4eae640 10 monclient: monmap: 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:58:20.651484+0000 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:58:20.651484+0000 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:58:20.631 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:20.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.651+0000 7fcce4eae640 1 Processor -- start 2024-03-29T10:58:20.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.651+0000 7fcce4eae640 1 -- start start 2024-03-29T10:58:20.632 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.651+0000 7fcce4eae640 10 monclient: init 2024-03-29T10:58:20.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.652+0000 7fcce4eae640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:20.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.652+0000 7fcce4eae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:20.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.652+0000 7fcce4eae640 10 monclient(hunting): picked mon.noname-a con 0x7fcce01422b0 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:20.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.652+0000 7fcce4eae640 10 monclient(hunting): picked mon.noname-c con 0x7fcce0142690 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:20.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.652+0000 7fcce4eae640 10 monclient(hunting): picked mon.noname-b con 0x7fcce00b17d0 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:20.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.652+0000 7fcce4eae640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce0142c80 con 0x7fcce01422b0 2024-03-29T10:58:20.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.652+0000 7fcce4eae640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce00aae40 con 0x7fcce0142690 2024-03-29T10:58:20.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.652+0000 7fcce4eae640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce013fbd0 con 0x7fcce00b17d0 2024-03-29T10:58:20.633 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.652+0000 7fcce4eae640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:20.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.652+0000 7fcce4eae640 10 monclient(hunting): authenticate will time out at 1513.893896s 2024-03-29T10:58:20.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccded9d640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fcce00b17d0 0x7fcce00b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:51152/0 (socket says 172.21.15.5:51152) 2024-03-29T10:58:20.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccde59c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fcce01422b0 0x7fcce013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46274/0 (socket says 172.21.15.5:46274) 2024-03-29T10:58:20.634 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccded9d640 1 -- 172.21.15.5:0/1486799409 learned_addr learned my addr 172.21.15.5:0/1486799409 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:20.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123533027 0 0) 0x7fcce0142c80 con 0x7fcce01422b0 2024-03-29T10:58:20.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:58:20.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:20.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:20.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 10 monclient(hunting): my global_id is 4575 2024-03-29T10:58:20.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcccc003520 con 0x7fcce01422b0 2024-03-29T10:58:20.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603636370 0 0) 0x7fcce013fbd0 con 0x7fcce00b17d0 2024-03-29T10:58:20.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:58:20.636 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 10 monclient(hunting): my global_id is 4682 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce0142c80 con 0x7fcce00b17d0 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.653+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817287926 0 0) 0x7fcce00aae40 con 0x7fcce0142690 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient(hunting): my global_id is 4504 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce013fbd0 con 0x7fcce0142690 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3986948735 0 0) 0x7fcccc003520 con 0x7fcce01422b0 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 >> v1:172.21.15.5:6790/0 conn(0x7fcce00b17d0 legacy=0x7fcce00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 >> v1:172.21.15.177:6789/0 conn(0x7fcce0142690 legacy=0x7fcce00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 1 monclient: found mon.noname-a 2024-03-29T10:58:20.637 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce0151940 con 0x7fcce01422b0 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: _finish_auth 0 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fccd0001690 con 0x7fcce01422b0 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: dump: 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:20.638 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fcce4eae640 5 monclient: authenticate success, global_id 4575 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fcce4eae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccd00017f0 con 0x7fcce01422b0 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 1 -- 172.21.15.5:0/1486799409 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fccd0001b80 con 0x7fcce01422b0 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fccddd9b640 10 monclient: dump: 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:20.639 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fcce4eae640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.654+0000 7fcce4eae640 4 set_mon_vals no callback set 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.655+0000 7fcce4eae640 10 monclient: shutdown 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.655+0000 7fcce4eae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.655+0000 7fcce4eae640 1 -- 172.21.15.5:0/1486799409 >> v1:172.21.15.5:6789/0 conn(0x7fcce01422b0 legacy=0x7fcce013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:20.640 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.655+0000 7fcce4eae640 1 -- 172.21.15.5:0/1486799409 shutdown_connections 2024-03-29T10:58:20.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.655+0000 7fcce4eae640 1 -- 172.21.15.5:0/1486799409 >> 172.21.15.5:0/1486799409 conn(0x7fcce0066a00 msgr2=0x7fcce0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:20.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.655+0000 7fcce4eae640 1 -- 172.21.15.5:0/1486799409 shutdown_connections 2024-03-29T10:58:20.641 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.655+0000 7fcce4eae640 1 -- 172.21.15.5:0/1486799409 wait complete. 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.656+0000 7fcce4eae640 10 monclient: build_initial_monmap 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.656+0000 7fcce4eae640 10 monclient: monmap: 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:20.648 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.656+0000 7fcce4eae640 1 librados: starting msgr at 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.656+0000 7fcce4eae640 1 librados: starting objecter 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.657+0000 7fcce4eae640 1 Processor -- start 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.657+0000 7fcce4eae640 1 -- start start 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.657+0000 7fcce4eae640 1 librados: setting wanted keys 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.657+0000 7fcce4eae640 1 librados: calling monclient init 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.657+0000 7fcce4eae640 10 monclient: init 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.657+0000 7fcce4eae640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.657+0000 7fcce4eae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.657+0000 7fcce4eae640 10 monclient(hunting): picked mon.noname-c con 0x7fcce00b17d0 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.657+0000 7fcce4eae640 10 monclient(hunting): picked mon.noname-b con 0x7fcce0142690 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fcce4eae640 10 monclient(hunting): picked mon.noname-a con 0x7fcce014e4a0 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fcce4eae640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce01418a0 con 0x7fcce014e4a0 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fcce4eae640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce00b7560 con 0x7fcce0142690 2024-03-29T10:58:20.649 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fcce4eae640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce012a520 con 0x7fcce00b17d0 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fcce4eae640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fcce4eae640 10 monclient(hunting): authenticate will time out at 1513.899285s 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccde59c640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fcce00b17d0 0x7fcce00b5780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:51528/0 (socket says 172.21.15.5:51528) 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccde59c640 1 -- 172.21.15.5:0/2550335977 learned_addr learned my addr 172.21.15.5:0/2550335977 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccd6ffd640 10 client.?.objecter ms_handle_connect 0x7fcce014e4a0 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccd6ffd640 10 client.?.objecter ms_handle_connect 0x7fcce00b17d0 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228173436 0 0) 0x7fcce01418a0 con 0x7fcce014e4a0 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccd6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccd6ffd640 10 monclient(hunting): my global_id is 4578 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.658+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc0004eb0 con 0x7fcce014e4a0 2024-03-29T10:58:20.650 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725939583 0 0) 0x7fcce012a520 con 0x7fcce00b17d0 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 10 monclient(hunting): my global_id is 4685 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce01418a0 con 0x7fcce00b17d0 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 10 client.?.objecter ms_handle_connect 0x7fcce0142690 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2976438282 0 0) 0x7fcce01418a0 con 0x7fcce00b17d0 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcce012a520 con 0x7fcce00b17d0 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fccd0001b00 con 0x7fcce00b17d0 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 10 monclient(hunting): dump: 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:20.651 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.659+0000 7fccd6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1541947056 0 0) 0x7fccc0004eb0 con 0x7fcce014e4a0 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcce01418a0 con 0x7fcce014e4a0 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fccd8001b60 con 0x7fcce014e4a0 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:20.652 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient(hunting): dump: 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1159231654 0 0) 0x7fcce012a520 con 0x7fcce00b17d0 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 >> v1:172.21.15.177:6789/0 conn(0x7fcce0142690 legacy=0x7fcce013c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 >> v1:172.21.15.5:6789/0 conn(0x7fcce014e4a0 legacy=0x7fcce01de830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:20.653 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 1 monclient: found mon.c 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce012a6b0 con 0x7fcce00b17d0 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fcce4eae640 5 monclient: authenticate success, global_id 4685 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fcce4eae640 10 monclient: _renew_subs 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fcce4eae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fcce4eae640 1 -- 172.21.15.5:0/2550335977 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce01411f0 con 0x7fcce00b17d0 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fcce4eae640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fcce4eae640 10 monclient: _renew_subs 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fcce4eae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fcce4eae640 1 -- 172.21.15.5:0/2550335977 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce0150d50 con 0x7fcce00b17d0 2024-03-29T10:58:20.654 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fcce4eae640 1 librados: init done 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccd0004a90 con 0x7fcce00b17d0 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fccd0004da0 con 0x7fcce00b17d0 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccd6ffd640 10 monclient: dump: 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:20.655 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.660+0000 7fccdd59a640 4 set_mon_vals no callback set 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.662+0000 7fcce4eae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.662+0000 7fcce4eae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.662+0000 7fcce4eae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.662+0000 7fcce4eae640 1 -- 172.21.15.5:0/2550335977 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc4005080 con 0x7fcce00b17d0 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.663+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fccd001ad70 con 0x7fcce00b17d0 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.663+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1216745598 0 0) 0x7fccd0084b40 con 0x7fcce00b17d0 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.663+0000 7fccd6ffd640 10 client.4685.objecter ms_dispatch 0x7fcce0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.663+0000 7fccd6ffd640 3 client.4685.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.663+0000 7fccd6ffd640 3 client.4685.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.663+0000 7fccd6ffd640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.664+0000 7fccd7fff640 10 monclient: get_auth_request con 0x7fccc006dee0 auth_method 0 2024-03-29T10:58:20.656 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.664+0000 7fccd6ffd640 10 client.4685.objecter ms_handle_connect 0x7fccc006dee0 2024-03-29T10:58:20.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.671+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fccd0050420 con 0x7fcce00b17d0 2024-03-29T10:58:20.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.671+0000 7fccd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:20.657 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:20.671+0000 7fccd6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:21.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.061+0000 7fcce4eae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:21.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.061+0000 7fcce4eae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:21.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.061+0000 7fcce4eae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:21.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.061+0000 7fcce4eae640 1 -- 172.21.15.5:0/2550335977 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccc4005330 con 0x7fcce00b17d0 2024-03-29T10:58:21.042 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.062+0000 7fccd6ffd640 1 -- 172.21.15.5:0/2550335977 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 16110640) 0x7fccd0052350 con 0x7fcce00b17d0 2024-03-29T10:58:21.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.062+0000 7fccd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:21.043 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.062+0000 7fccd6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:21.043 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:58:21.043 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:57:56.452759+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:56.452759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:21.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.065+0000 7fcce4eae640 10 librados: watch_flush enter 2024-03-29T10:58:21.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.065+0000 7fcce4eae640 10 librados: watch_flush exit 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.065+0000 7fcce4eae640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.065+0000 7fcce4eae640 10 client.4685.objecter successfully canceled tick 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.065+0000 7fcce4eae640 1 -- 172.21.15.5:0/2550335977 >> v1:172.21.15.5:6816/2847500151 conn(0x7fccc006dee0 legacy=0x7fccc00703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.065+0000 7fcce4eae640 10 monclient: shutdown 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.065+0000 7fcce4eae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.065+0000 7fcce4eae640 1 -- 172.21.15.5:0/2550335977 >> v1:172.21.15.5:6790/0 conn(0x7fcce00b17d0 legacy=0x7fcce00b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.065+0000 7fcce4eae640 1 -- 172.21.15.5:0/2550335977 shutdown_connections 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.065+0000 7fcce4eae640 1 -- 172.21.15.5:0/2550335977 >> 172.21.15.5:0/2550335977 conn(0x7fcce0066a00 msgr2=0x7fcce013e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.066+0000 7fcce4eae640 1 -- 172.21.15.5:0/2550335977 shutdown_connections 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.066+0000 7fcce4eae640 1 -- 172.21.15.5:0/2550335977 wait complete. 2024-03-29T10:58:21.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:21.066+0000 7fcce4eae640 1 librados: shutdown 2024-03-29T10:58:21.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:27.870 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:28.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.038+0000 7f4555b4d640 10 monclient: get_monmap_and_config 2024-03-29T10:58:28.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.038+0000 7f4555b4d640 10 monclient: build_initial_monmap 2024-03-29T10:58:28.019 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.038+0000 7f4555b4d640 10 monclient: monmap: 2024-03-29T10:58:28.019 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:28.019 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:28.020 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:58:28.039841+0000 2024-03-29T10:58:28.020 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:58:28.039841+0000 2024-03-29T10:58:28.020 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:28.020 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:28.020 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:28.020 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:58:28.020 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:58:28.020 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:28.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.041+0000 7f4555b4d640 1 Processor -- start 2024-03-29T10:58:28.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.041+0000 7f4555b4d640 1 -- start start 2024-03-29T10:58:28.021 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.041+0000 7f4555b4d640 10 monclient: init 2024-03-29T10:58:28.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.042+0000 7f4555b4d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:28.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.042+0000 7f4555b4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:28.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.042+0000 7f4554b4b640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f45480389e0 0x7f4548033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:49830/0 (socket says 172.21.15.5:49830) 2024-03-29T10:58:28.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.042+0000 7f4554b4b640 1 -- 172.21.15.5:0/1043144461 learned_addr learned my addr 172.21.15.5:0/1043144461 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:28.022 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.042+0000 7f4555b4d640 10 monclient(hunting): picked mon.noname-b con 0x7f45480389e0 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:28.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.043+0000 7f4555b4d640 10 monclient(hunting): picked mon.noname-a con 0x7f4548038dc0 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:28.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.043+0000 7f4555b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f4548030660 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:28.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.043+0000 7f4555b4d640 1 -- 172.21.15.5:0/1043144461 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45480c54c0 con 0x7f4548038dc0 2024-03-29T10:58:28.023 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.043+0000 7f4555b4d640 1 -- 172.21.15.5:0/1043144461 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4548036430 con 0x7f4548030660 2024-03-29T10:58:28.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.045+0000 7f4555b4d640 1 -- 172.21.15.5:0/1043144461 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4548039840 con 0x7f45480389e0 2024-03-29T10:58:28.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f4555b4d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:28.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f4555b4d640 10 monclient(hunting): authenticate will time out at 1521.287163s 2024-03-29T10:58:28.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 1 -- 172.21.15.5:0/1043144461 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966930835 0 0) 0x7f4548039840 con 0x7f45480389e0 2024-03-29T10:58:28.026 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:28.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:28.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:28.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient(hunting): my global_id is 4688 2024-03-29T10:58:28.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 1 -- 172.21.15.5:0/1043144461 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4540003520 con 0x7f45480389e0 2024-03-29T10:58:28.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 1 -- 172.21.15.5:0/1043144461 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 929472813 0 0) 0x7f4540003520 con 0x7f45480389e0 2024-03-29T10:58:28.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 1 -- 172.21.15.5:0/1043144461 >> v1:172.21.15.177:6789/0 conn(0x7f4548030660 legacy=0x7f45480d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:28.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 1 -- 172.21.15.5:0/1043144461 >> v1:172.21.15.5:6789/0 conn(0x7f4548038dc0 legacy=0x7f45480cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:28.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:58:28.027 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 1 -- 172.21.15.5:0/1043144461 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4548035690 con 0x7f45480389e0 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 1 -- 172.21.15.5:0/1043144461 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f453c001690 con 0x7f45480389e0 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.046+0000 7f454f7fe640 10 monclient: dump: 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:28.028 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.047+0000 7f4555b4d640 5 monclient: authenticate success, global_id 4688 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.047+0000 7f4555b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.047+0000 7f454f7fe640 1 -- 172.21.15.5:0/1043144461 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f453c001920 con 0x7f45480389e0 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.047+0000 7f454f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.047+0000 7f454f7fe640 1 -- 172.21.15.5:0/1043144461 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f453c001cb0 con 0x7f45480389e0 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.047+0000 7f454f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:28.029 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.047+0000 7f454f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.047+0000 7f454f7fe640 10 monclient: dump: 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:28.030 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:28.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.047+0000 7f4555b4d640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:28.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.047+0000 7f4555b4d640 4 set_mon_vals no callback set 2024-03-29T10:58:28.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.048+0000 7f4555b4d640 10 monclient: shutdown 2024-03-29T10:58:28.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.048+0000 7f4555b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:28.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.048+0000 7f4555b4d640 1 -- 172.21.15.5:0/1043144461 >> v1:172.21.15.5:6790/0 conn(0x7f45480389e0 legacy=0x7f4548033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.031 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.050+0000 7f4555b4d640 1 -- 172.21.15.5:0/1043144461 shutdown_connections 2024-03-29T10:58:28.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.050+0000 7f4555b4d640 1 -- 172.21.15.5:0/1043144461 >> 172.21.15.5:0/1043144461 conn(0x7f4548005c00 msgr2=0x7f4548006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:28.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.051+0000 7f4555b4d640 1 -- 172.21.15.5:0/1043144461 shutdown_connections 2024-03-29T10:58:28.032 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.051+0000 7f4555b4d640 1 -- 172.21.15.5:0/1043144461 wait complete. 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.052+0000 7f4555b4d640 10 monclient: build_initial_monmap 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.052+0000 7f4555b4d640 10 monclient: monmap: 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:28.033 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.052+0000 7f4555b4d640 1 librados: starting msgr at 2024-03-29T10:58:28.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.052+0000 7f4555b4d640 1 librados: starting objecter 2024-03-29T10:58:28.034 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.052+0000 7f4555b4d640 1 Processor -- start 2024-03-29T10:58:28.035 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.052+0000 7f4555b4d640 1 -- start start 2024-03-29T10:58:28.036 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.055+0000 7f4555b4d640 1 librados: setting wanted keys 2024-03-29T10:58:28.036 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.055+0000 7f4555b4d640 1 librados: calling monclient init 2024-03-29T10:58:28.036 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.055+0000 7f4555b4d640 10 monclient: init 2024-03-29T10:58:28.036 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.056+0000 7f4555b4d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:28.036 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.056+0000 7f4555b4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:28.036 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.056+0000 7f4555b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f4548030660 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:28.037 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.056+0000 7f4554b4b640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7f4548030660 0x7f45480c3230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:58478/0 (socket says 172.21.15.5:58478) 2024-03-29T10:58:28.040 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.060+0000 7f4554b4b640 1 -- 172.21.15.5:0/3756853802 learned_addr learned my addr 172.21.15.5:0/3756853802 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:28.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.060+0000 7f4555b4d640 10 monclient(hunting): picked mon.noname-b con 0x7f45480389e0 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:28.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.060+0000 7f454d7fa640 10 client.?.objecter ms_handle_connect 0x7f4548030660 2024-03-29T10:58:28.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.060+0000 7f454d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:28.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.060+0000 7f4555b4d640 10 monclient(hunting): picked mon.noname-a con 0x7f454802d150 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:28.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.060+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45480c7b70 con 0x7f454802d150 2024-03-29T10:58:28.041 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.060+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454802e090 con 0x7f45480389e0 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.063+0000 7f454d7fa640 10 client.?.objecter ms_handle_connect 0x7f45480389e0 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.063+0000 7f454d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.063+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445700401 0 0) 0x7f454802e090 con 0x7f45480389e0 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.063+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454802e300 con 0x7f4548030660 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f4555b4d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f4555b4d640 10 monclient(hunting): authenticate will time out at 1521.305149s 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 10 monclient(hunting): my global_id is 4510 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534004eb0 con 0x7f45480389e0 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 10 client.?.objecter ms_handle_connect 0x7f454802d150 2024-03-29T10:58:28.045 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669495972 0 0) 0x7f4534004eb0 con 0x7f45480389e0 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f454802e090 con 0x7f45480389e0 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f455004fdd0 con 0x7f45480389e0 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 10 monclient(hunting): dump: 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:28.046 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:28.047 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:28.047 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.064+0000 7f454d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:28.048 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123914971 0 0) 0x7f454802e300 con 0x7f4548030660 2024-03-29T10:58:28.048 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:28.048 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 10 monclient(hunting): my global_id is 4691 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534004eb0 con 0x7f4548030660 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 612543902 0 0) 0x7f454802e090 con 0x7f45480389e0 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 >> v1:172.21.15.5:6790/0 conn(0x7f4548030660 legacy=0x7f45480c3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 >> v1:172.21.15.5:6789/0 conn(0x7f454802d150 legacy=0x7f4548174db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 1 monclient: found mon.b 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.065+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480c7720 con 0x7f45480389e0 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f454d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f454d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:58:28.049 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f454d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f4555b4d640 5 monclient: authenticate success, global_id 4510 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f4555b4d640 10 monclient: _renew_subs 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f4555b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45480c78b0 con 0x7f45480389e0 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f4555b4d640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f4555b4d640 10 monclient: _renew_subs 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f4555b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45480cab30 con 0x7f45480389e0 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f4555b4d640 1 librados: init done 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f455004f400 con 0x7f45480389e0 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f454d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7f4550075f10 con 0x7f45480389e0 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f454d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f454d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:28.050 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f454d7fa640 10 monclient: dump: 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:28.051 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.066+0000 7f454effd640 4 set_mon_vals no callback set 2024-03-29T10:58:28.052 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.068+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7f4550067bf0 con 0x7f45480389e0 2024-03-29T10:58:28.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.069+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1216745598 0 0) 0x7f45500fe370 con 0x7f45480389e0 2024-03-29T10:58:28.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.069+0000 7f454d7fa640 10 client.4510.objecter ms_dispatch 0x7f4550000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:28.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.069+0000 7f454d7fa640 3 client.4510.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:28.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.069+0000 7f454d7fa640 3 client.4510.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:28.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.069+0000 7f454d7fa640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-29T10:58:28.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.069+0000 7f4555b4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:28.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.069+0000 7f4555b4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:28.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.069+0000 7f4555b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:58:28.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.069+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4530005080 con 0x7f45480389e0 2024-03-29T10:58:28.053 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.069+0000 7f4554b4b640 10 monclient: get_auth_request con 0x7f453406dab0 auth_method 0 2024-03-29T10:58:28.054 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.073+0000 7f454d7fa640 10 client.4510.objecter ms_handle_connect 0x7f453406dab0 2024-03-29T10:58:28.062 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.081+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45500c9c50 con 0x7f45480389e0 2024-03-29T10:58:28.062 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.081+0000 7f454d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:28.062 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.081+0000 7f454d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:28.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.420+0000 7f4555b4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:28.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.420+0000 7f4555b4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:28.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.420+0000 7f4555b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:58:28.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.420+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4530005330 con 0x7f45480389e0 2024-03-29T10:58:28.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.423+0000 7f454d7fa640 1 -- 172.21.15.5:0/3756853802 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 16110640) 0x7f45500cbb80 con 0x7f45480389e0 2024-03-29T10:58:28.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.423+0000 7f454d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:28.404 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.423+0000 7f454d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:28.404 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:58:28.404 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:57:56.452759+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:56.452759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:28.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.427+0000 7f4555b4d640 10 librados: watch_flush enter 2024-03-29T10:58:28.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 10 librados: watch_flush exit 2024-03-29T10:58:28.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-29T10:58:28.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 10 client.4510.objecter successfully canceled tick 2024-03-29T10:58:28.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 >> v1:172.21.15.5:6816/2847500151 conn(0x7f453406dab0 legacy=0x7f453406ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 10 monclient: shutdown 2024-03-29T10:58:28.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:28.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 >> v1:172.21.15.177:6789/0 conn(0x7f45480389e0 legacy=0x7f4548169570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 shutdown_connections 2024-03-29T10:58:28.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 >> 172.21.15.5:0/3756853802 conn(0x7f4548005c00 msgr2=0x7f4548034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:28.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 shutdown_connections 2024-03-29T10:58:28.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 1 -- 172.21.15.5:0/3756853802 wait complete. 2024-03-29T10:58:28.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:28.428+0000 7f4555b4d640 1 librados: shutdown 2024-03-29T10:58:28.423 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:35.231 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:35.398 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.417+0000 7ff7c1f6b640 10 monclient: get_monmap_and_config 2024-03-29T10:58:35.398 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.417+0000 7ff7c1f6b640 10 monclient: build_initial_monmap 2024-03-29T10:58:35.398 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.417+0000 7ff7c1f6b640 10 monclient: monmap: 2024-03-29T10:58:35.398 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:35.398 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:35.398 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:58:35.418927+0000 2024-03-29T10:58:35.398 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:58:35.418927+0000 2024-03-29T10:58:35.399 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:35.399 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:35.399 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:35.399 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:58:35.399 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:58:35.399 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:35.401 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.421+0000 7ff7c1f6b640 1 Processor -- start 2024-03-29T10:58:35.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.421+0000 7ff7c1f6b640 1 -- start start 2024-03-29T10:58:35.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.421+0000 7ff7c1f6b640 10 monclient: init 2024-03-29T10:58:35.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.422+0000 7ff7c1f6b640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:35.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.422+0000 7ff7c1f6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:35.402 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.422+0000 7ff7c1f6b640 10 monclient(hunting): picked mon.noname-a con 0x7ff7bc0b2790 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:35.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.422+0000 7ff7c1f6b640 10 monclient(hunting): picked mon.noname-c con 0x7ff7bc142b10 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.422+0000 7ff7c1f6b640 10 monclient(hunting): picked mon.noname-b con 0x7ff7bc0b31b0 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.422+0000 7ff7c1f6b640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc141f00 con 0x7ff7bc0b2790 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.422+0000 7ff7c1f6b640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc0aaf40 con 0x7ff7bc142b10 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.422+0000 7ff7c1f6b640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc0b3870 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.422+0000 7ff7c1f6b640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.422+0000 7ff7c1f6b640 10 monclient(hunting): authenticate will time out at 1528.664035s 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.423+0000 7ff7c176a640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7ff7bc0b31b0 0x7ff7bc14d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:35662/0 (socket says 172.21.15.5:35662) 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.423+0000 7ff7c176a640 1 -- 172.21.15.5:0/1051104461 learned_addr learned my addr 172.21.15.5:0/1051104461 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.424+0000 7ff7bb7fe640 1 -- 172.21.15.5:0/1051104461 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401536753 0 0) 0x7ff7bc0b3870 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.424+0000 7ff7bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.424+0000 7ff7bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.424+0000 7ff7bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.424+0000 7ff7bb7fe640 10 monclient(hunting): my global_id is 4694 2024-03-29T10:58:35.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.424+0000 7ff7bb7fe640 1 -- 172.21.15.5:0/1051104461 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a4003520 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.425+0000 7ff7bb7fe640 1 -- 172.21.15.5:0/1051104461 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2003950369 0 0) 0x7ff7a4003520 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.425+0000 7ff7bb7fe640 1 -- 172.21.15.5:0/1051104461 >> v1:172.21.15.177:6789/0 conn(0x7ff7bc142b10 legacy=0x7ff7bc13b8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:35.412 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 1 -- 172.21.15.5:0/1051104461 >> v1:172.21.15.5:6789/0 conn(0x7ff7bc0b2790 legacy=0x7ff7bc142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 1 monclient: found mon.noname-b 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 1 -- 172.21.15.5:0/1051104461 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc0b3a00 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 1 -- 172.21.15.5:0/1051104461 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff7ac001690 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 10 monclient: dump: 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:35.413 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7c1f6b640 5 monclient: authenticate success, global_id 4694 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7c1f6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.430+0000 7ff7bb7fe640 1 -- 172.21.15.5:0/1051104461 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7ac001860 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.431+0000 7ff7bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.431+0000 7ff7bb7fe640 1 -- 172.21.15.5:0/1051104461 <== mon.2 v1:172.21.15.5:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff7ac001ad0 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.431+0000 7ff7bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:35.414 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.431+0000 7ff7bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.431+0000 7ff7bb7fe640 10 monclient: dump: 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.431+0000 7ff7c1f6b640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.431+0000 7ff7c1f6b640 4 set_mon_vals no callback set 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.432+0000 7ff7c1f6b640 10 monclient: shutdown 2024-03-29T10:58:35.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.432+0000 7ff7c1f6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:35.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.432+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/1051104461 >> v1:172.21.15.5:6790/0 conn(0x7ff7bc0b31b0 legacy=0x7ff7bc14d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:35.416 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.433+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/1051104461 shutdown_connections 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.433+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/1051104461 >> 172.21.15.5:0/1051104461 conn(0x7ff7bc06d0e0 msgr2=0x7ff7bc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.433+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/1051104461 shutdown_connections 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.433+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/1051104461 wait complete. 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.434+0000 7ff7c1f6b640 10 monclient: build_initial_monmap 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.434+0000 7ff7c1f6b640 10 monclient: monmap: 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:35.417 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.435+0000 7ff7c1f6b640 1 librados: starting msgr at 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.435+0000 7ff7c1f6b640 1 librados: starting objecter 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.435+0000 7ff7c1f6b640 1 Processor -- start 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.435+0000 7ff7c1f6b640 1 -- start start 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.435+0000 7ff7c1f6b640 1 librados: setting wanted keys 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.435+0000 7ff7c1f6b640 1 librados: calling monclient init 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.435+0000 7ff7c1f6b640 10 monclient: init 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.436+0000 7ff7c1f6b640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.436+0000 7ff7c1f6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:35.418 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.436+0000 7ff7c1f6b640 10 monclient(hunting): picked mon.noname-a con 0x7ff7bc0b2790 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:35.422 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.440+0000 7ff7c1f6b640 10 monclient(hunting): picked mon.noname-b con 0x7ff7bc0b31b0 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:35.422 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.440+0000 7ff7c1f6b640 10 monclient(hunting): picked mon.noname-c con 0x7ff7bc142b10 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:35.422 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.440+0000 7ff7c1f6b640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc13edf0 con 0x7ff7bc0b2790 2024-03-29T10:58:35.422 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.440+0000 7ff7c1f6b640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc1d6240 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.422 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.440+0000 7ff7c1f6b640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc13f060 con 0x7ff7bc142b10 2024-03-29T10:58:35.422 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.440+0000 7ff7c1f6b640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:35.422 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.440+0000 7ff7c1f6b640 10 monclient(hunting): authenticate will time out at 1528.681236s 2024-03-29T10:58:35.422 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.440+0000 7ff7bbfff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7ff7bc0b31b0 0x7ff7bc1d66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:43947/0 (socket says 172.21.15.5:43947) 2024-03-29T10:58:35.423 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.440+0000 7ff7bbfff640 1 -- 172.21.15.5:0/341313362 learned_addr learned my addr 172.21.15.5:0/341313362 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:35.423 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.441+0000 7ff7b97fa640 10 client.?.objecter ms_handle_connect 0x7ff7bc0b31b0 2024-03-29T10:58:35.423 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.441+0000 7ff7b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:35.423 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.441+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964147224 0 0) 0x7ff7bc1d6240 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.423 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.441+0000 7ff7b97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:35.423 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.441+0000 7ff7b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:35.423 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.441+0000 7ff7b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:35.423 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.441+0000 7ff7b97fa640 10 monclient(hunting): my global_id is 4516 2024-03-29T10:58:35.423 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.441+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a8004eb0 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.425 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.442+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4152444966 0 0) 0x7ff7a8004eb0 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.425 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.442+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7bc1d6240 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.425 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.442+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff7b0001890 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.425 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.442+0000 7ff7b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:35.425 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.442+0000 7ff7b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:35.425 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.442+0000 7ff7b97fa640 10 monclient(hunting): dump: 2024-03-29T10:58:35.425 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:35.425 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:35.425 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:35.426 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:35.426 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:35.426 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:35.426 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:35.426 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:35.426 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:35.426 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:35.426 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.442+0000 7ff7b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:35.426 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.442+0000 7ff7b97fa640 10 client.?.objecter ms_handle_connect 0x7ff7bc142b10 2024-03-29T10:58:35.426 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.442+0000 7ff7b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:35.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 <== mon.1 v1:172.21.15.177:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3908177101 0 0) 0x7ff7bc1d6240 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 >> v1:172.21.15.5:6790/0 conn(0x7ff7bc142b10 legacy=0x7ff7bc135c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:35.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 >> v1:172.21.15.5:6789/0 conn(0x7ff7bc0b2790 legacy=0x7ff7bc13d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:35.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:58:35.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 1 monclient: found mon.b 2024-03-29T10:58:35.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:35.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:58:35.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc144230 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.428 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 10 monclient: _finish_auth 0 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.445+0000 7ff7b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 <== mon.1 v1:172.21.15.177:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7b00019f0 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 <== mon.1 v1:172.21.15.177:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7ff7b0003870 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7b97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7b97fa640 10 monclient: dump: 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:35.429 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7ba7fc640 4 set_mon_vals no callback set 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7c1f6b640 5 monclient: authenticate success, global_id 4516 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7c1f6b640 10 monclient: _renew_subs 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7c1f6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/341313362 --> v1:172.21.15.177:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7bc13deb0 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7c1f6b640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7c1f6b640 10 monclient: _renew_subs 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7c1f6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/341313362 --> v1:172.21.15.177:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7bc13d330 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.446+0000 7ff7c1f6b640 1 librados: init done 2024-03-29T10:58:35.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.449+0000 7ff7c1f6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:35.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.449+0000 7ff7c1f6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:35.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.449+0000 7ff7c1f6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:58:35.432 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.449+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/341313362 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7bc0b6c00 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.453+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 <== mon.1 v1:172.21.15.177:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7ff7b00039d0 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.453+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 <== mon.1 v1:172.21.15.177:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1216745598 0 0) 0x7ff7b0084a60 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.453+0000 7ff7b97fa640 10 client.4516.objecter ms_dispatch 0x7ff7bc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:35.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.453+0000 7ff7b97fa640 3 client.4516.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:35.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.453+0000 7ff7b97fa640 3 client.4516.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:35.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.453+0000 7ff7b97fa640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T10:58:35.434 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.454+0000 7ff7c0f69640 10 monclient: get_auth_request con 0x7ff7a806dc10 auth_method 0 2024-03-29T10:58:35.435 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.454+0000 7ff7b97fa640 10 client.4516.objecter ms_handle_connect 0x7ff7a806dc10 2024-03-29T10:58:35.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.458+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 <== mon.1 v1:172.21.15.177:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7b0050170 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.459+0000 7ff7b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:35.439 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.459+0000 7ff7b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:35.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.822+0000 7ff7c1f6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:35.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.822+0000 7ff7c1f6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:35.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.822+0000 7ff7c1f6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.177:6789/0 2024-03-29T10:58:35.803 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.822+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/341313362 --> v1:172.21.15.177:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7bc1d63f0 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.826+0000 7ff7b97fa640 1 -- 172.21.15.5:0/341313362 <== mon.1 v1:172.21.15.177:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 16110640) 0x7ff7b0052270 con 0x7ff7bc0b31b0 2024-03-29T10:58:35.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.826+0000 7ff7b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:35.807 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.826+0000 7ff7b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:35.807 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:58:35.807 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:57:56.452759+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:56.452759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:35.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 10 librados: watch_flush enter 2024-03-29T10:58:35.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 10 librados: watch_flush exit 2024-03-29T10:58:35.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T10:58:35.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 10 client.4516.objecter successfully canceled tick 2024-03-29T10:58:35.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/341313362 >> v1:172.21.15.5:6816/2847500151 conn(0x7ff7a806dc10 legacy=0x7ff7a80700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:35.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 10 monclient: shutdown 2024-03-29T10:58:35.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:35.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/341313362 >> v1:172.21.15.177:6789/0 conn(0x7ff7bc0b31b0 legacy=0x7ff7bc1d66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:35.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/341313362 shutdown_connections 2024-03-29T10:58:35.812 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/341313362 >> 172.21.15.5:0/341313362 conn(0x7ff7bc06d0e0 msgr2=0x7ff7bc0b8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:35.812 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.830+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/341313362 shutdown_connections 2024-03-29T10:58:35.812 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.831+0000 7ff7c1f6b640 1 -- 172.21.15.5:0/341313362 wait complete. 2024-03-29T10:58:35.812 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:35.831+0000 7ff7c1f6b640 1 librados: shutdown 2024-03-29T10:58:35.831 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:42.639 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:42.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.811+0000 7fea2e858640 10 monclient: get_monmap_and_config 2024-03-29T10:58:42.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.811+0000 7fea2e858640 10 monclient: build_initial_monmap 2024-03-29T10:58:42.792 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.811+0000 7fea2e858640 10 monclient: monmap: 2024-03-29T10:58:42.792 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:42.792 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:42.793 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:58:42.812565+0000 2024-03-29T10:58:42.793 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:58:42.812565+0000 2024-03-29T10:58:42.793 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:42.793 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:42.793 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:42.793 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:58:42.793 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:58:42.793 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:42.794 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.813+0000 7fea2e858640 1 Processor -- start 2024-03-29T10:58:42.794 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.813+0000 7fea2e858640 1 -- start start 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.813+0000 7fea2e858640 10 monclient: init 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.814+0000 7fea2e858640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.814+0000 7fea2e858640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.814+0000 7fea2e858640 10 monclient(hunting): picked mon.noname-b con 0x7fea280b0860 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.814+0000 7fea2e858640 10 monclient(hunting): picked mon.noname-c con 0x7fea2813ef30 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.814+0000 7fea2e858640 10 monclient(hunting): picked mon.noname-a con 0x7fea280af8e0 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.814+0000 7fea2e858640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea280b0c40 con 0x7fea280af8e0 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.814+0000 7fea2e858640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea280aed10 con 0x7fea2813ef30 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.814+0000 7fea2e858640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea280afd50 con 0x7fea280b0860 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.814+0000 7fea2e858640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:42.795 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.814+0000 7fea2e858640 10 monclient(hunting): authenticate will time out at 1536.055240s 2024-03-29T10:58:42.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.816+0000 7fea27fff640 1 --1- >> v1:172.21.15.5:6790/0 conn(0x7fea280b0860 0x7fea2813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6790/0 says I am v1:172.21.15.5:44968/0 (socket says 172.21.15.5:44968) 2024-03-29T10:58:42.796 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.816+0000 7fea27fff640 1 -- 172.21.15.5:0/691603852 learned_addr learned my addr 172.21.15.5:0/691603852 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:42.797 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.816+0000 7fea26ffd640 1 -- 172.21.15.5:0/691603852 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668344569 0 0) 0x7fea280afd50 con 0x7fea280b0860 2024-03-29T10:58:42.797 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.816+0000 7fea26ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:42.797 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.816+0000 7fea26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:42.797 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.816+0000 7fea26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:42.797 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.816+0000 7fea26ffd640 10 monclient(hunting): my global_id is 4700 2024-03-29T10:58:42.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.816+0000 7fea26ffd640 1 -- 172.21.15.5:0/691603852 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea10003520 con 0x7fea280b0860 2024-03-29T10:58:42.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 1 -- 172.21.15.5:0/691603852 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3110996711 0 0) 0x7fea10003520 con 0x7fea280b0860 2024-03-29T10:58:42.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 1 -- 172.21.15.5:0/691603852 >> v1:172.21.15.177:6789/0 conn(0x7fea2813ef30 legacy=0x7fea280af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:42.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 1 -- 172.21.15.5:0/691603852 >> v1:172.21.15.5:6789/0 conn(0x7fea280af8e0 legacy=0x7fea28149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:42.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:42.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 1 monclient: found mon.noname-b 2024-03-29T10:58:42.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:42.798 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.5:6790/0 2024-03-29T10:58:42.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 1 -- 172.21.15.5:0/691603852 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea280ab1b0 con 0x7fea280b0860 2024-03-29T10:58:42.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:42.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:42.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:42.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea2e858640 5 monclient: authenticate success, global_id 4700 2024-03-29T10:58:42.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea2e858640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:42.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 1 -- 172.21.15.5:0/691603852 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fea18001690 con 0x7fea280b0860 2024-03-29T10:58:42.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:42.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:42.799 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 10 monclient: dump: 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea2e858640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 1 -- 172.21.15.5:0/691603852 <== mon.2 v1:172.21.15.5:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea18001920 con 0x7fea280b0860 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea2e858640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:42.800 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.817+0000 7fea2e858640 4 set_mon_vals no callback set 2024-03-29T10:58:42.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.819+0000 7fea2e858640 10 monclient: shutdown 2024-03-29T10:58:42.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.819+0000 7fea2e858640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:42.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.819+0000 7fea2e858640 1 -- 172.21.15.5:0/691603852 >> v1:172.21.15.5:6790/0 conn(0x7fea280b0860 legacy=0x7fea2813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:42.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.820+0000 7fea2e858640 1 -- 172.21.15.5:0/691603852 shutdown_connections 2024-03-29T10:58:42.801 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.820+0000 7fea2e858640 1 -- 172.21.15.5:0/691603852 >> 172.21.15.5:0/691603852 conn(0x7fea2806b660 msgr2=0x7fea2806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:42.802 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.822+0000 7fea2e858640 1 -- 172.21.15.5:0/691603852 shutdown_connections 2024-03-29T10:58:42.805 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.825+0000 7fea2e858640 1 -- 172.21.15.5:0/691603852 wait complete. 2024-03-29T10:58:42.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.826+0000 7fea2e858640 10 monclient: build_initial_monmap 2024-03-29T10:58:42.806 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.826+0000 7fea2e858640 10 monclient: monmap: 2024-03-29T10:58:42.806 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:42.807 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:42.807 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:58:42.807 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:58:42.807 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:42.807 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:42.807 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:42.807 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:58:42.807 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:58:42.807 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:42.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.826+0000 7fea2e858640 1 librados: starting msgr at 2024-03-29T10:58:42.808 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.826+0000 7fea2e858640 1 librados: starting objecter 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.826+0000 7fea2e858640 1 Processor -- start 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.827+0000 7fea2e858640 1 -- start start 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.827+0000 7fea2e858640 1 librados: setting wanted keys 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.827+0000 7fea2e858640 1 librados: calling monclient init 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.827+0000 7fea2e858640 10 monclient: init 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.828+0000 7fea2e858640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.828+0000 7fea2e858640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.828+0000 7fea2e858640 10 monclient(hunting): picked mon.noname-a con 0x7fea280af8e0 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.828+0000 7fea2e858640 10 monclient(hunting): picked mon.noname-c con 0x7fea280b0860 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.828+0000 7fea2e858640 10 monclient(hunting): picked mon.noname-b con 0x7fea2813ef30 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.828+0000 7fea2e858640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2806f5e0 con 0x7fea280af8e0 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.828+0000 7fea2e858640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea280b4310 con 0x7fea2813ef30 2024-03-29T10:58:42.810 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.828+0000 7fea2e858640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea280b3e80 con 0x7fea280b0860 2024-03-29T10:58:42.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.828+0000 7fea2e858640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:42.811 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.828+0000 7fea2e858640 10 monclient(hunting): authenticate will time out at 1536.069375s 2024-03-29T10:58:42.812 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.832+0000 7fea27fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fea280af8e0 0x7fea281e7360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35337/0 (socket says 172.21.15.5:35337) 2024-03-29T10:58:42.812 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.832+0000 7fea27fff640 1 -- 172.21.15.5:0/1810997808 learned_addr learned my addr 172.21.15.5:0/1810997808 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:42.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.832+0000 7fea24ff9640 10 client.?.objecter ms_handle_connect 0x7fea280af8e0 2024-03-29T10:58:42.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.832+0000 7fea24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:42.815 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.832+0000 7fea24ff9640 10 client.?.objecter ms_handle_connect 0x7fea280b0860 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.832+0000 7fea24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194797614 0 0) 0x7fea2806f5e0 con 0x7fea280af8e0 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 10 monclient(hunting): my global_id is 4587 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea14004eb0 con 0x7fea280af8e0 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264659314 0 0) 0x7fea280b3e80 con 0x7fea280b0860 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 10 monclient(hunting): my global_id is 4703 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2806f5e0 con 0x7fea280b0860 2024-03-29T10:58:42.816 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 10 client.?.objecter ms_handle_connect 0x7fea2813ef30 2024-03-29T10:58:42.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.833+0000 7fea24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:42.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401426941 0 0) 0x7fea280b4310 con 0x7fea2813ef30 2024-03-29T10:58:42.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:58:42.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:42.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:42.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 10 monclient(hunting): my global_id is 4519 2024-03-29T10:58:42.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea280b3e80 con 0x7fea2813ef30 2024-03-29T10:58:42.817 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 117307018 0 0) 0x7fea280b3e80 con 0x7fea2813ef30 2024-03-29T10:58:42.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea280b4310 con 0x7fea2813ef30 2024-03-29T10:58:42.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fea20001890 con 0x7fea2813ef30 2024-03-29T10:58:42.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:42.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:42.822 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 10 monclient(hunting): dump: 2024-03-29T10:58:42.822 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:42.822 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.834+0000 7fea24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081251216 0 0) 0x7fea2806f5e0 con 0x7fea280b0860 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea280b3e80 con 0x7fea280b0860 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fea1c001640 con 0x7fea280b0860 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:42.823 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 10 monclient(hunting): dump: 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4156921498 0 0) 0x7fea14004eb0 con 0x7fea280af8e0 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea2806f5e0 con 0x7fea280af8e0 2024-03-29T10:58:42.824 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fea18004ab0 con 0x7fea280af8e0 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.835+0000 7fea24ff9640 10 monclient(hunting): dump: 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.2 v1:172.21.15.5:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1703607603 0 0) 0x7fea280b3e80 con 0x7fea280b0860 2024-03-29T10:58:42.825 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 >> v1:172.21.15.177:6789/0 conn(0x7fea2813ef30 legacy=0x7fea281ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 >> v1:172.21.15.5:6789/0 conn(0x7fea280af8e0 legacy=0x7fea281e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 1 monclient: found mon.c 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.5:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea280b4010 con 0x7fea280b0860 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 10 monclient: _finish_auth 0 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea2e858640 5 monclient: authenticate success, global_id 4703 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea2e858640 10 monclient: _renew_subs 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea2e858640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea2e858640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.5:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2806d9d0 con 0x7fea280b0860 2024-03-29T10:58:42.826 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea2e858640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea2e858640 10 monclient: _renew_subs 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea2e858640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea2e858640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.5:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea280b2740 con 0x7fea280b0860 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.836+0000 7fea2e858640 1 librados: init done 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.837+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.2 v1:172.21.15.5:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea1c001b50 con 0x7fea280b0860 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.837+0000 7fea24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.837+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.2 v1:172.21.15.5:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fea1c003790 con 0x7fea280b0860 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.837+0000 7fea24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.837+0000 7fea24ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.837+0000 7fea24ff9640 10 monclient: dump: 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:42.827 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:42.828 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:42.828 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:42.828 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:42.828 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:42.828 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:42.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.837+0000 7fea25ffb640 4 set_mon_vals no callback set 2024-03-29T10:58:42.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.839+0000 7fea2e858640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:42.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.839+0000 7fea2e858640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:42.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.839+0000 7fea2e858640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:42.828 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.839+0000 7fea2e858640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea08005080 con 0x7fea280b0860 2024-03-29T10:58:42.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.843+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.2 v1:172.21.15.5:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fea1c019760 con 0x7fea280b0860 2024-03-29T10:58:42.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.844+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.2 v1:172.21.15.5:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1216745598 0 0) 0x7fea1c0835d0 con 0x7fea280b0860 2024-03-29T10:58:42.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.844+0000 7fea27fff640 10 monclient: get_auth_request con 0x7fea1406e210 auth_method 0 2024-03-29T10:58:42.829 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.844+0000 7fea24ff9640 10 client.4703.objecter ms_dispatch 0x7fea28000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:42.830 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.844+0000 7fea24ff9640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:42.830 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.844+0000 7fea24ff9640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:42.830 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.844+0000 7fea24ff9640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-29T10:58:42.830 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.844+0000 7fea24ff9640 10 client.4703.objecter ms_handle_connect 0x7fea1406e210 2024-03-29T10:58:42.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.851+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.2 v1:172.21.15.5:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea1c04eeb0 con 0x7fea280b0860 2024-03-29T10:58:42.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.851+0000 7fea24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:42.831 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:42.851+0000 7fea24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.227+0000 7fea2e858640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.227+0000 7fea2e858640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.227+0000 7fea2e858640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.5:6790/0 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.227+0000 7fea2e858640 1 -- 172.21.15.5:0/1810997808 --> v1:172.21.15.5:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea08005330 con 0x7fea280b0860 2024-03-29T10:58:43.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.229+0000 7fea24ff9640 1 -- 172.21.15.5:0/1810997808 <== mon.2 v1:172.21.15.5:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 16110640) 0x7fea1c050de0 con 0x7fea280b0860 2024-03-29T10:58:43.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.229+0000 7fea24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:43.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.229+0000 7fea24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:43.210 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:58:43.210 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:57:56.452759+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:56.452759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:43.214 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.234+0000 7fea2e858640 10 librados: watch_flush enter 2024-03-29T10:58:43.214 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.234+0000 7fea2e858640 10 librados: watch_flush exit 2024-03-29T10:58:43.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.234+0000 7fea2e858640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-29T10:58:43.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.234+0000 7fea2e858640 10 client.4703.objecter successfully canceled tick 2024-03-29T10:58:43.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.234+0000 7fea2e858640 1 -- 172.21.15.5:0/1810997808 >> v1:172.21.15.5:6816/2847500151 conn(0x7fea1406e210 legacy=0x7fea140706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:43.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.234+0000 7fea2e858640 10 monclient: shutdown 2024-03-29T10:58:43.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.234+0000 7fea2e858640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:43.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.234+0000 7fea2e858640 1 -- 172.21.15.5:0/1810997808 >> v1:172.21.15.5:6790/0 conn(0x7fea280b0860 legacy=0x7fea281e7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:43.215 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.235+0000 7fea2e858640 1 -- 172.21.15.5:0/1810997808 shutdown_connections 2024-03-29T10:58:43.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.235+0000 7fea2e858640 1 -- 172.21.15.5:0/1810997808 >> 172.21.15.5:0/1810997808 conn(0x7fea2806b660 msgr2=0x7fea2806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:43.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.235+0000 7fea2e858640 1 -- 172.21.15.5:0/1810997808 shutdown_connections 2024-03-29T10:58:43.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.235+0000 7fea2e858640 1 -- 172.21.15.5:0/1810997808 wait complete. 2024-03-29T10:58:43.216 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:43.235+0000 7fea2e858640 1 librados: shutdown 2024-03-29T10:58:43.236 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:50.043 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:50.179 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.199+0000 7fefb017c640 10 monclient: get_monmap_and_config 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.199+0000 7fefb017c640 10 monclient: build_initial_monmap 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.199+0000 7fefb017c640 10 monclient: monmap: 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:58:50.200621+0000 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:58:50.200621+0000 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-b 2024-03-29T10:58:50.180 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:50.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.200+0000 7fefb017c640 1 Processor -- start 2024-03-29T10:58:50.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.200+0000 7fefb017c640 1 -- start start 2024-03-29T10:58:50.181 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.200+0000 7fefb017c640 10 monclient: init 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.201+0000 7fefb017c640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.201+0000 7fefb017c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.201+0000 7fefb017c640 10 monclient(hunting): picked mon.noname-a con 0x7fefa81422b0 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.201+0000 7fefb017c640 10 monclient(hunting): picked mon.noname-b con 0x7fefa8142690 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.201+0000 7fefb017c640 10 monclient(hunting): picked mon.noname-c con 0x7fefa80b17d0 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.202+0000 7fefb017c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa8142c80 con 0x7fefa81422b0 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.202+0000 7fefb017c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa80aae40 con 0x7fefa80b17d0 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.202+0000 7fefb017c640 1 -- --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa813fbd0 con 0x7fefa8142690 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.202+0000 7fefb017c640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.202+0000 7fefb017c640 10 monclient(hunting): authenticate will time out at 1543.443126s 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.202+0000 7fefadf18640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fefa81422b0 0x7fefa813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:41208/0 (socket says 172.21.15.5:41208) 2024-03-29T10:58:50.183 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.202+0000 7fefadf18640 1 -- 172.21.15.5:0/1716647703 learned_addr learned my addr 172.21.15.5:0/1716647703 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:50.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834001511 0 0) 0x7fefa8142c80 con 0x7fefa81422b0 2024-03-29T10:58:50.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient(hunting): _negotiate 2024-03-29T10:58:50.185 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient(hunting): my global_id is 4590 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef94003520 con 0x7fefa81422b0 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276197080 0 0) 0x7fefa813fbd0 con 0x7fefa8142690 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient(hunting): _negotiate 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient(hunting): my global_id is 4706 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef94003840 con 0x7fefa8142690 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1595727727 0 0) 0x7fef94003520 con 0x7fefa81422b0 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 >> v1:172.21.15.5:6790/0 conn(0x7fefa8142690 legacy=0x7fefa80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 >> v1:172.21.15.177:6789/0 conn(0x7fefa80b17d0 legacy=0x7fefa80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient: _finish_hunting 0 2024-03-29T10:58:50.186 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 1 monclient: found mon.noname-a 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.5:6789/0 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa8151940 con 0x7fefa81422b0 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient: _finish_auth 0 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient: _check_auth_tickets 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fefa4001670 con 0x7fefa81422b0 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefacf16640 10 monclient: dump: 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.187 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefb017c640 5 monclient: authenticate success, global_id 4590 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.203+0000 7fefb017c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.204+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 <== mon.0 v1:172.21.15.5:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefa40017d0 con 0x7fefa81422b0 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.204+0000 7fefacf16640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.204+0000 7fefacf16640 1 -- 172.21.15.5:0/1716647703 <== mon.0 v1:172.21.15.5:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fefa4001b60 con 0x7fefa81422b0 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.204+0000 7fefacf16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.204+0000 7fefacf16640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.204+0000 7fefacf16640 10 monclient: dump: 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:50.188 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.204+0000 7fefb017c640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.204+0000 7fefb017c640 4 set_mon_vals no callback set 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.205+0000 7fefb017c640 10 monclient: shutdown 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.205+0000 7fefb017c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.205+0000 7fefb017c640 1 -- 172.21.15.5:0/1716647703 >> v1:172.21.15.5:6789/0 conn(0x7fefa81422b0 legacy=0x7fefa813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.205+0000 7fefb017c640 1 -- 172.21.15.5:0/1716647703 shutdown_connections 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.205+0000 7fefb017c640 1 -- 172.21.15.5:0/1716647703 >> 172.21.15.5:0/1716647703 conn(0x7fefa8066a00 msgr2=0x7fefa8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:50.189 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.205+0000 7fefb017c640 1 -- 172.21.15.5:0/1716647703 shutdown_connections 2024-03-29T10:58:50.190 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.205+0000 7fefb017c640 1 -- 172.21.15.5:0/1716647703 wait complete. 2024-03-29T10:58:50.196 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.206+0000 7fefb017c640 10 monclient: build_initial_monmap 2024-03-29T10:58:50.196 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.206+0000 7fefb017c640 10 monclient: monmap: 2024-03-29T10:58:50.196 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-03-29T10:58:50.196 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:50.196 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-03-29T10:58:50.196 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-03-29T10:58:50.196 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.noname-b 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.noname-c 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.207+0000 7fefb017c640 1 librados: starting msgr at 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.207+0000 7fefb017c640 1 librados: starting objecter 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.207+0000 7fefb017c640 1 Processor -- start 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.207+0000 7fefb017c640 1 -- start start 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.207+0000 7fefb017c640 1 librados: setting wanted keys 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.207+0000 7fefb017c640 1 librados: calling monclient init 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.207+0000 7fefb017c640 10 monclient: init 2024-03-29T10:58:50.197 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.208+0000 7fefb017c640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.208+0000 7fefb017c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.209+0000 7fefb017c640 10 monclient(hunting): picked mon.noname-c con 0x7fefa80b17d0 addr v1:172.21.15.5:6790/0 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.209+0000 7fefb017c640 10 monclient(hunting): picked mon.noname-b con 0x7fefa81422b0 addr v1:172.21.15.177:6789/0 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.209+0000 7fefad717640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fefa81422b0 0x7fefa814d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.5:41524/0 (socket says 172.21.15.5:41524) 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.209+0000 7fefad717640 1 -- 172.21.15.5:0/2977698410 learned_addr learned my addr 172.21.15.5:0/2977698410 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.209+0000 7fefb017c640 10 monclient(hunting): picked mon.noname-a con 0x7fefa8142690 addr v1:172.21.15.5:6789/0 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.209+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa81513c0 con 0x7fefa8142690 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.209+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa814e220 con 0x7fefa81422b0 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.209+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa80ac850 con 0x7fefa80b17d0 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.209+0000 7fefb017c640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.209+0000 7fefb017c640 10 monclient(hunting): authenticate will time out at 1543.450836s 2024-03-29T10:58:50.198 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 client.?.objecter ms_handle_connect 0x7fefa81422b0 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 client.?.objecter ms_handle_connect 0x7fefa80b17d0 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958135596 0 0) 0x7fefa814e220 con 0x7fefa81422b0 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 monclient(hunting): my global_id is 4525 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef8c004eb0 con 0x7fefa81422b0 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 client.?.objecter ms_handle_connect 0x7fefa8142690 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.2 v1:172.21.15.5:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950685478 0 0) 0x7fefa80ac850 con 0x7fefa80b17d0 2024-03-29T10:58:50.199 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 10 monclient(hunting): my global_id is 4709 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.210+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa814e220 con 0x7fefa80b17d0 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230875560 0 0) 0x7fefa81513c0 con 0x7fefa8142690 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 10 monclient(hunting): my global_id is 4593 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa80ac850 con 0x7fefa8142690 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4060572270 0 0) 0x7fefa80ac850 con 0x7fefa8142690 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefa81513c0 con 0x7fefa8142690 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fefa0001bf0 con 0x7fefa8142690 2024-03-29T10:58:50.200 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 10 monclient(hunting): dump: 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:50.201 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.2 v1:172.21.15.5:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 756202954 0 0) 0x7fefa814e220 con 0x7fefa80b17d0 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.211+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefa80ac850 con 0x7fefa80b17d0 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2341406603 0 0) 0x7fef8c004eb0 con 0x7fefa81422b0 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.177:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefa814e220 con 0x7fefa81422b0 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.2 v1:172.21.15.5:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fefa4001b60 con 0x7fefa80b17d0 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient(hunting): dump: 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:50.202 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fef98001910 con 0x7fefa81422b0 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient(hunting): dump: 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:50.203 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:50.204 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:50.204 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:50.204 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:50.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.0 v1:172.21.15.5:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2117486128 0 0) 0x7fefa81513c0 con 0x7fefa8142690 2024-03-29T10:58:50.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 >> v1:172.21.15.5:6790/0 conn(0x7fefa80b17d0 legacy=0x7fefa814f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 >> v1:172.21.15.177:6789/0 conn(0x7fefa81422b0 legacy=0x7fefa814d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:50.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 monclient: found mon.a 2024-03-29T10:58:50.204 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:50.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:58:50.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa80acb30 con 0x7fefa8142690 2024-03-29T10:58:50.205 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fefb017c640 5 monclient: authenticate success, global_id 4593 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fefb017c640 10 monclient: _renew_subs 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fefb017c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa80ace10 con 0x7fefa8142690 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fefb017c640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fefb017c640 10 monclient: _renew_subs 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fefb017c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa81510c0 con 0x7fefa8142690 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fefb017c640 1 librados: init done 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.0 v1:172.21.15.5:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefa0003970 con 0x7fefa8142690 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:50.206 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.0 v1:172.21.15.5:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3231003908 0 0) 0x7fefa0002100 con 0x7fefa8142690 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9e7fc640 10 monclient: dump: 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:fsid fcc4bb0a-729e-4ad6-a413-fad605d2bb4a 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:created 2024-03-29T10:47:15.323549+0000 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.a 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.177:6789/0 mon.b 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:2: v1:172.21.15.5:6790/0 mon.c 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr: 2024-03-29T10:58:50.207 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.212+0000 7fef9ffff640 4 set_mon_vals no callback set 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.0 v1:172.21.15.5:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1763045371 0 0) 0x7fefa0019af0 con 0x7fefa8142690 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fefb017c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fefb017c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fefb017c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.0 v1:172.21.15.5:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1216745598 0 0) 0x7fefa0083ad0 con 0x7fefa8142690 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fef9e7fc640 10 client.4593.objecter ms_dispatch 0x7fefa8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fef9e7fc640 3 client.4593.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fef9e7fc640 3 client.4593.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef90005080 con 0x7fefa8142690 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fefadf18640 10 monclient: get_auth_request con 0x7fef8c0769f0 auth_method 0 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.215+0000 7fef9e7fc640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-29T10:58:50.208 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.216+0000 7fef9e7fc640 10 client.4593.objecter ms_handle_connect 0x7fef8c0769f0 2024-03-29T10:58:50.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.221+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.0 v1:172.21.15.5:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefa004f1e0 con 0x7fefa8142690 2024-03-29T10:58:50.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.221+0000 7fef9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:50.209 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.221+0000 7fef9e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.586+0000 7fefb017c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.586+0000 7fefb017c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.586+0000 7fefb017c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.5:6789/0 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.586+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 --> v1:172.21.15.5:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef90005330 con 0x7fefa8142690 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.587+0000 7fef9e7fc640 1 -- 172.21.15.5:0/2977698410 <== mon.0 v1:172.21.15.5:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 16110640) 0x7fefa00512e0 con 0x7fefa8142690 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.587+0000 7fef9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:50.568 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.587+0000 7fef9e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:50.568 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-29T10:58:50.568 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"fcc4bb0a-729e-4ad6-a413-fad605d2bb4a","created":"2024-03-29T10:47:51.529606+0000","modified":"2024-03-29T10:57:56.452759+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:56.452759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6a5633b9-e2cf-4b21-aed0-cbbd0abeaf28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"79b602b0-e74b-4fa1-8418-09e0e3a40335","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5e501eca-51c0-4012-a928-1985451ebaca","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"867c7e71-84e2-4983-8560-f9f830e2c1ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f9bc5cd6-e0e0-49a4-b63e-e5f247fcedb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8b7c8ae-e849-48e3-b698-eddd18808d4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f863ddfb-54f0-4042-8f43-5355132f26af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59ef6382-f411-4a44-9e59-59ad158f625b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:50.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 10 librados: watch_flush enter 2024-03-29T10:58:50.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 10 librados: watch_flush exit 2024-03-29T10:58:50.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-29T10:58:50.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 10 client.4593.objecter successfully canceled tick 2024-03-29T10:58:50.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 >> v1:172.21.15.5:6816/2847500151 conn(0x7fef8c0769f0 legacy=0x7fef8c078eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.571 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 10 monclient: shutdown 2024-03-29T10:58:50.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:50.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 >> v1:172.21.15.5:6789/0 conn(0x7fefa8142690 legacy=0x7fefa814db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 shutdown_connections 2024-03-29T10:58:50.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 >> 172.21.15.5:0/2977698410 conn(0x7fefa8066a00 msgr2=0x7fefa80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:50.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 shutdown_connections 2024-03-29T10:58:50.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.591+0000 7fefb017c640 1 -- 172.21.15.5:0/2977698410 wait complete. 2024-03-29T10:58:50.572 INFO:teuthology.misc.health.smithi005.stderr:2024-03-29T10:58:50.592+0000 7fefb017c640 1 librados: shutdown 2024-03-29T10:58:50.586 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:50.587 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:58:50.588 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:58:50.589 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:58:50.589 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:50.620 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:58:50.620 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:58:50.620 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:50.690 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:58:50.690 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:58:50.690 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:50.760 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:58:50.760 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:58:50.760 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:50.830 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:58:50.830 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:58:50.831 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:50.865 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:58:50.866 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:58:50.866 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:50.937 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:58:50.937 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:58:50.938 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:51.002 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:58:51.002 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:58:51.002 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:51.076 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:58:51.076 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:58:51.076 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:58:51.076 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:51.101 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:58:51.101 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:58:51.101 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:51.142 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:58:51.142 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:58:51.142 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:58:51.142 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:51.154 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:58:51.155 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:58:51.155 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:51.207 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:58:51.207 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:58:51.207 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:51.218 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:58:51.218 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:58:51.219 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:58:51.400 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T10:58:51.401 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:58:52.188 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T10:58:52.188 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:58:52.276 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T10:58:52.276 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:58:52.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T10:58:52.335 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:58:52.391 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T10:58:52.392 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:58:56.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T10:58:56.140 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:58:56.230 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T10:58:56.231 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:58:56.305 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T10:58:56.305 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:58:56.363 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:58:56.371 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656/data/mon.a.tgz 2024-03-29T10:58:56.372 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-03-29T10:58:56.390 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.VcHxuN2Abr 2024-03-29T10:58:56.390 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.VcHxuN2Abr 2024-03-29T10:58:56.510 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.VcHxuN2Abr 2024-03-29T10:58:56.600 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.VcHxuN2Abr is 211KB 2024-03-29T10:58:56.838 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.VcHxuN2Abr 2024-03-29T10:58:56.854 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656/data/mon.c.tgz 2024-03-29T10:58:56.854 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-03-29T10:58:56.909 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.SEgBGATcP9 2024-03-29T10:58:56.909 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.SEgBGATcP9 2024-03-29T10:58:57.011 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.SEgBGATcP9 2024-03-29T10:58:57.096 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.SEgBGATcP9 is 211KB 2024-03-29T10:58:57.189 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.SEgBGATcP9 2024-03-29T10:58:57.203 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656/data/mon.b.tgz 2024-03-29T10:58:57.203 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-03-29T10:58:57.221 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.EulRgoJOzL 2024-03-29T10:58:57.222 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.EulRgoJOzL 2024-03-29T10:58:57.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.EulRgoJOzL 2024-03-29T10:58:57.426 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.EulRgoJOzL is 211KB 2024-03-29T10:58:57.518 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.EulRgoJOzL 2024-03-29T10:58:57.533 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:58:57.534 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:58:57.536 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:58:57.617 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:58:57.657 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:58:57.657 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:58:57.659 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656/remote/smithi005/crash 2024-03-29T10:58:57.660 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:58:57.687 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656/remote/smithi177/crash 2024-03-29T10:58:57.688 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:58:57.724 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:58:57.724 DEBUG:teuthology.orchestra.run.smithi005:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:58:57.731 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:58:57.755 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40305.log 2024-03-29T10:58:57.755 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40890.log 2024-03-29T10:58:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:gzip/var/log/ceph/ceph-client.admin.40305.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41046.log 2024-03-29T10:58:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40890.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:58:57.790 INFO:teuthology.orchestra.run.smithi005.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40305.log.gz 2024-03-29T10:58:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41046.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40890.log.gz 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39676.log 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38998.log 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41046.log.gz 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40383.log 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39598.log 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39676.log.gz 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41397.log 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38998.log.gz 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39598.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40383.log.gz 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38842.log 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40149.log 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39598.log.gz 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41202.log 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41397.log.gz 2024-03-29T10:58:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39325.log 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38842.log.gz 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41202.log: gzip 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40149.log.gz -5 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39559.log 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39115.log 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41202.log.gz 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39037.log 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39325.log.gz 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39115.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39559.log.gz 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41007.log 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34839.log 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39037.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39115.log.gz 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38411.log 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39037.log.gz 2024-03-29T10:58:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.34839.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41007.log.gz 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38294.log 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40266.log 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34839.log.gz 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38920.log 86.0% 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38411.log.gz 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40578.log 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38294.log.gz 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40422.log 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40266.log.gz 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40578.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38920.log.gz 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40106.log 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38372.log 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40578.log.gz 2024-03-29T10:58:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40067.log 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40422.log.gz 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40106.log.gz 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39520.log 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40734.log 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38372.log.gz 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39520.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40067.log.gz -5 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40929.log 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39520.log.gz 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41358.log 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41553.log 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40734.log.gz 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41358.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40929.log.gz 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40500.log 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi005.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41358.log.gz 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41553.log: /var/log/ceph/ceph-client.admin.40500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41475.log 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41553.log.gz 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40773.log 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40500.log.gz 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40227.log 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41475.log.gz 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38686.log 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40773.log.gz 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38764.log 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-03-29T10:58:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38686.log: gzip -5 --verbose -- 85.7% /var/log/ceph/ceph-client.admin.38881.log -- replaced with /var/log/ceph/ceph-client.admin.40227.log.gz 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41631.log 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38686.log.gz 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38881.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38764.log.gz 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40695.log 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41631.log: gzip -5 --verbose -- 86.7% /var/log/ceph/ceph-client.admin.39442.log -- replaced with /var/log/ceph/ceph-client.admin.38881.log.gz 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40027.log 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41631.log.gz 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39442.log: gzip -5 87.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40695.log.gz 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr: /var/log/ceph/ceph-osd.1.log 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39364.log 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39442.log.gz 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41241.log 2024-03-29T10:58:57.796 INFO:teuthology.orchestra.run.smithi005.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40027.log.gz 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41241.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39364.log.gz 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40968.log 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39286.log: gzip -5 --verbose -- 86.1% /var/log/ceph/ceph-client.admin.41280.log 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41241.log.gz 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40851.log 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38803.log 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40851.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40968.log.gz 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38569.log 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.41280.log.gz 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38803.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40851.log.gz 2024-03-29T10:58:57.797 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39832.log 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38333.log 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38803.log.gz 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41085.log 86.4% 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38569.log.gz 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39832.log: /var/log/ceph/ceph-client.admin.38333.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28740.log 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39832.log.gz 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41709.log 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38333.log.gz 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39754.log 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.tmp-client.admin.28740.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28740.log.gz 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41085.log.gz 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39637.log 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39167.log 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39754.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41709.log.gz 2024-03-29T10:58:57.798 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39754.log.gz 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39247.log 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39637.log.gz 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38450.log 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39247.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39167.log.gz 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41163.log 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38450.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39247.log.gz 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41163.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38450.log.gz 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38608.log 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41163.log.gz 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41670.log 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.38608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38959.log 2024-03-29T10:58:57.799 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41514.log 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38608.log.gz 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41592.log 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41670.log.gz 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39076.log 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38959.log.gz 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39481.log 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41514.log.gz 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41319.log 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39481.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41592.log.gz 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39076.log.gz 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40188.log 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41319.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39481.log.gz 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39207.log 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41319.log.gz 2024-03-29T10:58:57.800 INFO:teuthology.orchestra.run.smithi005.stderr:gzip/var/log/ceph/ceph-client.admin.40188.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39403.log 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41436.log 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39403.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40188.log.gz 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39988.log 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39207.log.gz 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41436.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39403.log.gz 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40539.log 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41436.log.gz 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40812.log 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39988.log: /var/log/ceph/ceph-client.admin.40539.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40812.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40656.log 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39988.log.gz 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40539.log.gz 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40344.log 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2024-03-29T10:58:57.801 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38647.log 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40344.log: gzip -5 --verbose -- 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40656.log.gz 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr: /var/log/ceph/ceph-client.admin.39715.log 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38647.log: gzip 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40344.log.gz 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38530.log 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39715.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38647.log.gz 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34942.log 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39715.log.gz 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38489.log 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.34942.log: 85.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38725.log 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38530.log.gz 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38489.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34942.log.gz 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40461.log 2024-03-29T10:58:57.802 INFO:teuthology.orchestra.run.smithi005.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38489.log.gz/var/log/ceph/ceph-client.admin.38725.log: 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40461.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38725.log.gz 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40461.log.gz 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34548.log 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.34548.log: /var/log/ceph/ceph-osd.2.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34548.log.gz 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40617.log 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39793.log 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40617.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40617.log.gz 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39949.log 2024-03-29T10:58:57.803 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39871.log 2024-03-29T10:58:57.804 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39949.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39793.log.gz 2024-03-29T10:58:57.804 INFO:teuthology.orchestra.run.smithi005.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39949.log.gz 2024-03-29T10:58:57.804 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:58:57.804 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39871.log: /var/log/ceph/ceph-mon.c.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39871.log.gz 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34458.log 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34618.log 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34658.log 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34618.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.34458.log.gz -5 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34658.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34618.log.gz 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34698.log 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34658.log.gz 2024-03-29T10:58:57.810 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34698.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34738.log 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34698.log.gz 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34738.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34738.log.gz 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34538.log 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34498.log 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34538.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34538.log.gz 2024-03-29T10:58:57.811 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34498.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34498.log.gz 2024-03-29T10:58:57.812 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34578.log 2024-03-29T10:58:57.812 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34578.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34578.log.gz 2024-03-29T10:58:57.812 INFO:teuthology.orchestra.run.smithi177.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:58:57.853 INFO:teuthology.orchestra.run.smithi005.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:59:10.046 INFO:teuthology.orchestra.run.smithi177.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:59:10.174 INFO:teuthology.orchestra.run.smithi177.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:59:10.269 INFO:teuthology.orchestra.run.smithi005.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:59:14.016 INFO:teuthology.orchestra.run.smithi005.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T10:59:14.320 INFO:teuthology.orchestra.run.smithi177.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:59:14.350 INFO:teuthology.orchestra.run.smithi177.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:59:14.493 INFO:teuthology.orchestra.run.smithi005.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:59:15.849 INFO:teuthology.orchestra.run.smithi005.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:59:50.752 INFO:teuthology.orchestra.run.smithi005.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:59:52.677 INFO:teuthology.orchestra.run.smithi005.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:59:52.679 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-29T10:59:52.680 INFO:teuthology.orchestra.run.smithi005.stderr:real 0m54.937s 2024-03-29T10:59:52.680 INFO:teuthology.orchestra.run.smithi005.stderr:user 2m45.733s 2024-03-29T10:59:52.680 INFO:teuthology.orchestra.run.smithi005.stderr:sys 0m5.480s 2024-03-29T11:00:04.268 INFO:teuthology.orchestra.run.smithi177.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:00:04.270 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-29T11:00:04.270 INFO:teuthology.orchestra.run.smithi177.stderr:real 1m6.487s 2024-03-29T11:00:04.270 INFO:teuthology.orchestra.run.smithi177.stderr:user 1m59.489s 2024-03-29T11:00:04.270 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m4.495s 2024-03-29T11:00:04.270 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:00:04.271 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656/remote/smithi005/log 2024-03-29T11:00:04.272 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:00:27.297 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656/remote/smithi177/log 2024-03-29T11:00:27.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:00:46.415 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:00:46.607 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=77638ac0c5ff4ea0af69f7cdfdda2dfc Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:00:46.611 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:00:46.620 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:00:46.649 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:00:46.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:00:46.663 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:00:46.749 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:00:46.757 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:00:46.758 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:00:46.760 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:00:46.774 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:00:46.778 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:00:46.780 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:00:46.780 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-03-29T11:00:46.780 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +426us[ +428us] +/- 78ms 2024-03-29T11:00:46.780 INFO:teuthology.orchestra.run.smithi005.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -176us[ -175us] +/- 58ms 2024-03-29T11:00:46.781 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 -193us[ -192us] +/- 89ms 2024-03-29T11:00:46.781 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:00:46.782 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:00:46.782 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-29T11:00:46.783 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 +435us[ +425us] +/- 78ms 2024-03-29T11:00:46.783 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -193us[ -202us] +/- 58ms 2024-03-29T11:00:46.783 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -194us[ -204us] +/- 89ms 2024-03-29T11:00:46.783 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:00:46.785 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:00:46.793 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:00:46.794 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:00:46.816 DEBUG:teuthology.orchestra.run.smithi005:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:00:46.889 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:00:46.961 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:00:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:00:46.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:00:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:00:47.025 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:00:47.035 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:00:47.069 INFO:teuthology.task.internal:Duration was 1333.756375 seconds 2024-03-29T11:00:47.070 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:00:47.100 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:00:47.100 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:00:47.103 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:00:47.146 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:00:47.149 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:00:47.387 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:00:47.387 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-03-29T11:00:47.388 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:00:47.407 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-03-29T11:00:47.408 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:00:47.436 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:00:47.436 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:00:47.450 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:00:47.505 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:00:47.505 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:00:47.519 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:00:47.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:00:47.651 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:00:47.652 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:00:47.679 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:00:47.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:00:47.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:00:47.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:00:47.744 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-03-29T11:00:47.771 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-03-29T11:00:47.795 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:00:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:00:47.817 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:00:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:00:47.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:00:47.860 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:00:47.861 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656/remote/smithi005 2024-03-29T11:00:47.861 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:00:47.897 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656/remote/smithi177 2024-03-29T11:00:47.897 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:00:47.931 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:00:47.932 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:00:47.934 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:00:47.981 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:00:47.993 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:00:47.993 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:00:48.017 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:00:48.017 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:00:48.020 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:00:48.031 INFO:teuthology.orchestra.run.smithi005.stdout: 265835 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:00 /home/ubuntu/cephtest 2024-03-29T11:00:48.035 INFO:teuthology.orchestra.run.smithi177.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:00 /home/ubuntu/cephtest 2024-03-29T11:00:48.036 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:00:48.051 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:00:48.118 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:00:48.144 DEBUG:teuthology.nuke:shortname: smithi005 2024-03-29T11:00:48.144 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:00:48.171 DEBUG:teuthology.nuke:shortname: smithi177 2024-03-29T11:00:48.171 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:00:48.175 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:30:37.880496', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzB2nmx/jDNQL6ad35f2kCeylwaq4F/KlMlDoy0UNN5CCNeOuTbKQ3wP8tds7fvlFZfqRPeNnBzFN4JMvVnTcM='} 2024-03-29T11:00:48.197 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622656', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:30:37.879140', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcegcc0yDoT9QN1F1Kn+v508+jLs+9MnP5xfCQMWLaSXylIex7wd+H/dWc3zhPQGh+3o98lkHn9JWeyby1HAZo='} 2024-03-29T11:00:48.219 INFO:teuthology.orchestra.console.smithi005:Power off 2024-03-29T11:00:48.219 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:00:48.242 DEBUG:teuthology.orchestra.console.smithi005:power off output: Chassis Power Control: Down/Off 2024-03-29T11:00:48.242 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:48.258 INFO:teuthology.orchestra.console.smithi177:Power off 2024-03-29T11:00:48.258 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:00:48.268 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-03-29T11:00:48.283 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-03-29T11:00:48.283 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:48.306 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-29T11:00:52.269 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:52.307 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:52.380 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-03-29T11:00:52.417 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-29T11:00:56.384 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:56.418 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:56.495 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2024-03-29T11:00:56.527 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-29T11:00:56.596 INFO:teuthology.orchestra.console.smithi005:Power off completed 2024-03-29T11:01:00.531 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:01:00.643 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-03-29T11:01:00.743 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-03-29T11:01:00.848 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1333.7563750743866 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=77638ac0c5ff4ea0af69f7cdfdda2dfc status: fail success: false 2024-03-29T11:01:00.848 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:01:00.926 INFO:teuthology.run:FAIL